Python Eventlet + Javascript Websocket returns Blob - javascript

I am experimenting with Python's Eventlet Websocket support, using this simple example :
import eventlet
from eventlet import wsgi
from eventlet import websocket
from eventlet.support import six
# demo app
import os
import json
import time
import random
#websocket.WebSocketWSGI
def handle(ws):
""" This is the websocket handler function. Note that we
can dispatch based on path in here, too."""
if ws.path == '/echo':
while True:
ws.send("hello")
#ws.send(json.dumps({"msg":"hello"}))
time.sleep(1)
elif ws.path == '/data':
for i in six.moves.range(10000):
ws.send("0 %s %s\n" % (i, random.random()))
eventlet.sleep(0.1)
def dispatch(environ, start_response):
""" This resolves to the web page or the websocket depending on
the path."""
if environ['PATH_INFO'] == '/data':
return handle(environ, start_response)
else:
start_response('200 OK', [('content-type', 'text/html')])
return [open(os.path.join(
os.path.dirname(__file__),
'websocket.html')).read()]
if __name__ == "__main__":
# run an example app from the command line
listener = eventlet.listen(('127.0.0.1', 7000))
print("\nVisit http://localhost:7000/ in your websocket-capable browser.\n")
wsgi.server(listener, dispatch)
I'm not going to include the entire websocket handler I have in the Javascript, just the ws.onmessage method:
ws.onmessage = function (evt)
{
console.log(evt.data)
var received_msg = evt.data;
#Do stuff, i.e var obj = JSON.parse(received_msg)
#callback(obj)
};
The console.log(evt.data) indicates a succesful connection with the websocket (and you can assume this is all fine). However, the logging shows Blob {size: 31, type: ""} as the content of evt.data. I assume this is some kind of response object which is interpreted as binary (file) data (though I might be entirely wrong), but I'm not sure what to do with this.
I see that Blob data is often the data type for file-like objects. I suppose I could approach it as such, but I really only want to send json data back and forth. I've tried dumping a dict as a JSON and sending that, but it did the same. Even a string is received in Blob format.
How do I use eventlet for json data transmission?

As of 2017-05 Eventlet websocket API does not support string websocket frames. Patches are welcome, it's easy.
Your options:
read Blob at javascript end FileReader.readAsText
use another websocket library (pure Python implementation will work fine with Eventlet patching)
add string frame support to Eventlet websocket library

Related

Data posted to flask endpoint from JS not processed in endpoint

I have written a simple todo app with react acting as a frontend and flask handling CRUD from a DB. The app is using axios to handle the requests; GET completes fine however when attempting to POST JSON the flask api returns a 400 error. Here's some condensed sample code.
JS POST function.
function testPost(){
axios.post('http://'+window.location.hostname+':8000/todo/', {
title: "test123",
}).then(res => {
console.log(res)
}).catch(err => {
console.log(err)
})
}
Serverside
class Todo(Resource):
def post(self): # create a new todo
conn = pool.getconn()
cur = conn.cursor()
app.logger.info(request.form['title'])
cur.execute("INSERT INTO todo (task, done) VALUES (%s, %s)", (request.form['title'], False))
conn.commit()
app.logger.error(e)
cur.close()
pool.putconn(conn)
Other methods not shown
Then the rest of the server code attaching the resource to the api and the CORS setup (not shown in file order)
app = Flask(__name__)
CORS(app, methods=['POST','GET','PUT','DELETE'])
api = Api(app)
api.add_resource(Todo, '/todo/')
app.run(debug = True, host='0.0.0.0', port=port)
Tests
Using python to test the api works fine, running this in a seperate python file will add to the DB.
response = requests.post(URL + "todo/", data={"title": f"test{randint(1, 100)}"})
My best guess is that axios is not adding the data to the request in a way that the backend is unable to process. Before using axios I tried to make the request with XMLHttprequest however this presented the same problem. I swapped to axios on the recommendation of someone else, given its alleged improved simplicity.
request.form['key'] and request.get_json()['key'] are completely different fields python requests in the way I used it posts to the former and js posts to the latter. Modifying the function to use whichever is available fixes this.

Flask socketio multithread dump dictionary to queue and send to clients

Goal is simply to be able to make a thread queue of dictionaries and report them to client.
EDIT
This is different of Flask throwing 'working outside of request context' when starting sub thread because:
It is not done in a route function, it is done in socketio.start_background_task
The only socketio code takes place in context, with the socketio.emit we are sending a dictionary.
Strategy:
There are 2 different taks to perform in server side, for each build a thread, then in another socketio thread collect the results which are in a thread safe queue FIFO of dictionaries.
Then send these dictionaries to client and wait for each acknowledge.
So now the issue is reduced to solve:
RuntimeError: Working outside of request context.
from flask import Flask, flash, request, redirect, render_template, Response, escape, jsonify, url_for, session, copy_current_request_context
#socketio
from flask_socketio import SocketIO, send, emit, join_room, leave_room, close_room, rooms, disconnect
import threading
from threading import Thread, Event, Lock
import queue
import random
def ack(value):
if value != 'pong':
logger.info('unexpected return value')
def fn_i():
global q
while True:
time.sleep(1)
q.put({'key_i':random.random()})
return q
def fn_ii():
global q
while True:
time.sleep(10)
q.put({'key_ii':random.random()})
return q
app = Flask(__name__)
socketio = SocketIO(app, async_mode=async_mode)
thread1=None
thread2=None
collector_thread=None
q = queue.Queue()
thread_lock = Lock()
def background_thread_collector():
global thread1
global thread2
global q
thread1 = threading.Thread(target=fn_i)
thread1.start()
thread2 = threading.Thread(target=fn_ii)
thread2.start()
"""Example of how to send server generated events to clients."""
while True:
time.sleep(0.2)
while not q.empty():
socketio.emit('my_response',
q.get(), #{'data': 'Server generated event', 'count': count},
namespace='/test',
broadcast=True,
callback=ack
)
#app.route('/')
def index():
return render_template('index.html', async_mode=socketio.async_mode)
#socketio.on('connect', namespace='/test')
def test_connect():
global collector_thread
logger.info(' Client connected ' + request.sid)
with thread_lock:
if collector_thread is None:
collector_thread = socketio.start_background_task(background_thread_collector)
emit('my_response', {'data': 'Connected', 'count': 0})
if __name__ == '__main__':
socketio.run(app,
host='localhost',
port=10000,
debug=False) #True sends some exceptions and stops)
Cheers
This should be handled better by Flask-SocketIO, but the problem is that you are trying to use a callback on an emit that is set to broadcast to all clients:
socketio.emit('my_response',
q.get(), #{'data': 'Server generated event', 'count': count},
namespace='/test',
broadcast=True,
callback=ack
)
Remove the callback and the emit should work just fine.
Using threading queues with Flasksocketio is not straightforward because requires to handle the apps context among other things, for purpose of refreshing a server log file in client side found it was easier to simply use javascript in this case and the file can be updated accordingly. Even the former code with suggested alterations was not working because of apps context and nevertheless there are multiple blogs or stackoverflow which approach subject in none have found a complete working solution except implementing like explain because any other answer requires complete code which is working, and hence since able to implement here considering this is the accepted answer, Cheers.

socketio: data loss due to socket getting closed due to transport error

I've been trying very hard to figure this out but no luck so far.
So I am sending some data from server to client using socket in a loop however the client is unable to receive all the data and closes the socket in the middle of data transmission for no reason.
As you can see in the image below client successfully receives data till 11th iteration of the loop(refer to the server code below) however after that socket is closed cause of transport error. What possibly could I be doing wrong here?
Client side logs
Server side logs
(Python) server side code
from flask import Flask, render_template, request, flash, redirect, jsonify, make_response
from flask_socketio import SocketIO, emit, disconnect
import time
from gevent import monkey
monkey.patch_all()
app = Flask(__name__)
socketio = SocketIO(app, engineio_logger=True)
#socketio.on('run_tgt')
def run_tg(tg_args):
for x in range(20):
time.sleep(2)
emit('tg_output',x)
if __name__ == "__main__":
socketio.run(app, host='0.0.0.0', debug=True)
(Javascript) client side code
<script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.0.3/socket.io.js"></script>
var socket = io.connect('http://' + document.domain + ':' + location.port );
function run_tg() {
socket.emit('run_tgt', { 'data': 'data'});
socket.on('tg_output', function(tg_output) {
console.log(tg_output);
$("#output_div").append(tg_output);
});
}
Fixed the issue by adding monkey patch to the top of the script
from gevent import monkey
monkey.patch_all()
Also set async_handlers to true
socketio = SocketIO(app, async_handlers=True, engineio_logger=True)

Why does Node.js HTTP server not respond to a request from Python?

I've got a working HTTP node.js server.
I then created a program on python that uses the socket module to connect to the above server
Please for the time being do not mind the try and except statements. The code's connectTO() function simply connects to a server like any other code, with the exception that it handles some errors. Then the program send the message "hello". Next in the while loop it repeatedly waits for an answer and when it receives one, it prints it.
When I connect to the Node.js http server from python, I do get the message:
"You have just succesfully connected to the node.js server"
Which if you look at my code means that the s.connect(()) command was successful. My problem is that when a request is send to the server, it's supposed to output a message back, but it doesn't.
I also tried sending a message to the server, in which case the server sends back the following message:
HTTP/1.1 400 Bad Request
So why is the server not responding to the requests? Why is it rejecting them?
Python Client:
from socket import AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR
import threading, socket, time, sys
s = socket.socket(AF_INET,SOCK_STREAM)
def connectTO(host,port):
connect = False
count = 0
totalCount = 0
while connect!= True:
try:
s.connect((host,port))
connect = True
print("You have just succesfully connected to the node.js server")
except OSError:
count += 1
totalCount += 1
if totalCount == 40 and count == 4:
print("Error: 404. Connection failed repeatedly")
sys.exit(0)
elif count == 4:
print("Connection failed, retrying...")
count = 0
else:
pass
connectTO("IP_OF_NODE.jS_SERVER_GOES_HERE",777)
message = "hello"
s.send(message.encode("utf-8"))
while True:
try:
data, addr = s.recvfrom(1024)
if data == "":
pass
else:
print(data.decode())
except ConnectionResetError:
print("it seems like we can't reach the server anymore..")
print("This could be due to a change in your internet connection or the server.")
s.close()
Node.js HTTP server:
function onRequest(req, res) {
var postData = "";
var pathname = url.parse(req.url).pathname;
//Inform console of event recievent
console.log("Request for "+pathanme+" received.");
//Set the encoding to equivelant one used in html
req.setEncoding("utf8");
//add a listener for whenever info comes in and output full result
req.addListener("data", function(postDataChunk) {
postData += postDataChunk;
console.log("Received POST data chunk: '"+postDataChunk+"'");
});
req.addListener("end", function() {
route(handle, pathname, res, frontPage, postData);
});
};
http.createServer(onRequest).listen(port,ip);
console.log("Server has started.");
Some of my Research
I should also note that after some research, it seems that an HTTP server accepts HTTP requests, but I don't understand most of what's on Wikipedia. Is this the reason why the server is not responding? And how do I fix that while still using the socket module.
Also there are a lot of similar questions on Stack Overflow, but none help me solve my problem. One of them describes my issue, and the only answer is about "handshakes". Google is also pointless here, but from what I understand it is simply a reaction between the server and the client which defines what the protocol will be. Could this be what I'm missing, and how do I implement it?
Some of these questions also use modules that I'm not ready to use yet like websocket. Either that or they describe a way in which the server connects to the client, which can be done by directly calling python code or connecting to it from Node.js express. I want the client to be the one connecting to an HTTP server, by the means of the socket module in python. For the sake of future visitors who are looking for something like this, here are some of these question:
How to connect node.js app with python script?
Python Client to nodeJS Server with Socket.IO
Python connecting to an HTTP server
A blog that also does something similar to what is described above: https://www.sohamkamani.com/blog/2015/08/21/python-nodejs-comm/
Here is an answer that doesn't actually seem that obvious, but also solves the issue with only the relevant code. People who don't yet no much about servers in general will have probably missed it:
how to use socket fetch webpage use python
You will need to construct an HTTP request.
Example: GET / HTTP/1.1\n\n
Try this:
from socket import AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR
import threading, socket, time, sys
s = socket.socket(AF_INET,SOCK_STREAM)
def connectTO(host,port):
connect = False
count = 0
totalCount = 0
while connect!= True:
try:
s.connect((host,port))
connect = True
print("You have just succesfully connected to the node.js server")
except OSError:
count += 1
totalCount += 1
if totalCount == 40 and count == 4:
print("Error: 404. Connection failed repeatedly")
sys.exit(0)
elif count == 4:
print("Connection failed, retrying...")
count = 0
else:
pass
connectTO("IP_OF_NODE.jS_SERVER_GOES_HERE",777)
message = "GET / HTTP/1.1\n\n"
s.send(message.encode("utf-8"))
while True:
try:
data, addr = s.recvfrom(1024)
if data == "":
pass
else:
print(data.decode())
except ConnectionResetError:
print("it seems like we can't reach the server anymore..")
print("This could be due to a change in your internet connection or the server.")
s.close()
Read this to learn more about HTTP.
Now, I would recommend using this python lib to do what you're trying to do. It makes things much easier. However, if you are 100% set on using raw sockets, then you should make the node server use raw sockets as well. (Assuming you will only be connecting via python). Here is an excellent tutorial

Sending data from JavaScript to Python function locally with AJAX

I am trying to build a website where a user can enter text, which will be picked up via javascript, and sent to a python function where it will be posted to twitter. For the time being, the python function is being stored locally, along with the rest of the site. However, my AJAX isn't too great and I'm having a few issues.
I have written AJAX code which sends a POST request to the python function with the tweet, and the response is the entire python script. No connection is made to the socket my script is listening to. Below is the AJAX function and the python script. Any ideas what's going on?
Thanks in advance for any help!
$(function(){
$('#PostTweet').on('click', function(e) {
var tweet = document.getElementById("theTweet").value;
var len = tweet.length;
if(len > 140){
window.alert("Tweet too long. Please remove some characters");
}else{
callPython(tweet);
}
});
});
function callPython(tweet){
window.alert("sending");
$.ajax({
type: "POST",
url: "tweet.py",
data: tweet,
success: function(response){
window.alert(response);
}
})
}
And the Python Script:
from OAuthSettings import settings
import twitter
from socket import *
consumer_key = settings['consumer_key']
consumer_secret = settings['consumer_secret']
access_token_key = settings['access_token_key']
access_token_secret = settings['access_token_secret']
s = socket()
s.bind(('', 9999))
s.listen(4)
(ns, na) = s.accept()
def PostToTwits(data):
try:
api = twitter.Api(
consumer_key = consumer_key,
consumer_secret = consumer_secret,
access_token_key = access_token_key,
access_token_secret = access_token_secret)
api.PostUpdate(data)
makeConnection(s)
except twitter.TwitterError:
print 'Post Unsuccessful. Error Occurred'
def makeConnection(s):
while True:
print "connected with: " + str(na)
try:
data = ns.recv(4096)
print data
PostToTwits(data)
except:
ns.close()
s.close()
break
makeConnection(s)
Your problem is that you are working with pure sockets which know nothing about HTTP protocol. Take a look at Flask or Bottle web micro frameworks to see how to turn python script or function into web endpoint.
you need a webserver so that your can make request via web browser.
you can web framework like flask or django or you can use webpy.
A simple example using webpy from their website
import web
urls = (
'/(.*)', 'hello'
)
app = web.application(urls, globals())
class hello:
def GET(self, name):
if not name:
name = 'World'
return 'Hello, ' + name + '!'
if __name__ == "__main__":
app.run()
then you call url(your python function) from javascript.
You can totally write a simple web server using sockets, and indeed you've done so. But this approach will quickly get tedious for anything beyond a simple exercise.
For example, your code is restricted to handling a single request handler, which goes to the heart of your problem.
The url on the post request is wrong. In your setup there is no notion of a url "tweet.py". That url would actually work if you were also serving the web page where the jquery lives from the same server (but you can't be).
You have to post to "http://localhost:9999" and you can have any path you want after:"http://localhost:9999/foo", "http://localhost:9999/boo". Just make sure you run the python script from the command line first, so the server is listening.
Also the difference between a get and a post request is part of the HTTP protocol which your simple server doesn't know anything about. This mainly means that it doesn't matter what verb you use on the ajax request. Your server listens for all HTTP verb types.
Lastly, I'm not seeing any data being returned to the client. You need to do something like ns.sendall("Some response"). Tutorials for building a simple http server abound and show different ways of sending responses.

Categories

Resources