Develop Reference

How to pass data between Django module/app functions without using database in asynchronous web service

I've got a web service under development that uses Django and Django Channels to send data across websockets to a remote application. The arrangement is asynchronous and I pass information between the 2 by sending JSON formatted commands across websockets and then receive replies back on the same websocket.
The problem I'm having is figuring out how to get the replies back to a Javascript call from a Django template that invokes a Python function to initiate the JSON websocket question. Since the command question & data reply happen in different Django areas and the originating Javascript/Python functions call does not have a blocking statement, the Q&A are basically disconnected and I can't figure out how to get the results back to the browser.
Right now, my idea is to use Django global variables or store the results in the Django models. I can get either to work, but I beleive the Django global variables would not scale beyond multiple workers from runserver or if the system was eventually spread across multiple servers.
But since the reply data is for different purposes (for example, list of users waiting in a remote lobby, current debugging levels in remote system, etc), the database option seems unworkable because the reply data is varying structure. That, plus the replies are temporal and don't need to be permanently stored in the database.
Here's some code showing the flow. I'm open to different implementation recommendations or a direct answer to the question of how to share information between the 2 Django functions.
In the template, for testing, I just have a button defined like this:
<button id="request_lobby">Request Lobby</button>
With a Javascript function. This function is incomplete as I've yet to do anything about the response (because I can't figure out how to connect it):
$("#request_lobby").click(function(){
$.ajax({
type: "POST",
url: "{% url 'test_panel_function' %}",
data: { csrfmiddlewaretoken: '{{ csrf_token }}', button:"request_lobby" },
success: function(response){
}
});
});
This is the Django/Python function in views.py . The return channel for the remote application is pre-stored in the database as srv.server_channel when the websocket is initially connected (not shown):
#login_required
def test_panel_function(request):
button = request.POST.get('button', '')
if button == "request_lobby" :
srv = Server.objects.get(server_key="1234567890")
json_res = []
json_res.append({"COMMAND": "REQUESTLOBBY"})
message = ({
"text": json.dumps(json_res)
})
Channel(srv.server_channel).send(message)
return HttpResponse(button)
Later, the remote application sends the reply back on the websocket and it's received by a Django Channels demultiplexer in routing.py :
class RemoteDemultiplexer(WebsocketDemultiplexer):
mapping = {
"gLOBBY" : "gLOBBY.receive",
}
http_user = True
slight_ordering = True
channel_routing = [
route_class(RemoteDemultiplexer, path=r"^/server/(?P<server_key>[a-zA-Z0-9]+)$"),
route("gLOBBY.receive" , command_LOBBY),
]
And the consumer.py :
#channel_session
def command_LOBBY(message):
skey = message.channel_session["server_key"]
for x in range(int(message.content['LOBBY'])):
logger.info("USERNAME: " + message.content[str(x)]["USERNAME"])
logger.info("LOBBY_ID: " + message.content[str(x)]["LOBBY_ID"])
logger.info("OWNER_ID: " + message.content[str(x)]["IPADDRESS"])
logger.info("DATETIME: " + message.content[str(x)]["DATETIME"])
So I need to figure out how to get the reply data in command_LOBBY to the Javascript/Python function call in test_panel_function
Current ideas, both of which seem bad and why I think I need to ask this question for SO:
1) Use Django global variables:
Define in globals.py:
global_async_result = {}
And include in all relevant Django modules:
from test.globals import global_async_result
In order to make this work, when I originate the initial command in test_panel_function to send to the remote application (the REQUESTLOBBY), I'll include a randomized key in the JSON message which would be round-tripped back to command_LOBBY and then global_async_result dictionary would be indexed with the randomized key.
In test_panel_function , I would wait in a loop checking a flag for the results to be ready in global_async_result and then retrieve them from the randomized key and delete the entry in global_async_result.
Then the reply can be given back to the Javascript in the Django template.
That all makes sense to me, but uses global variables (bad), and seems that it wouldn't scale as the web service is spread across servers.
2) Store replies in Django mySQL model.py table
I could create a table in models.py to hold the replies temporarily. Since Django doesn't allow for dynamic or temporary table creations on the fly, this would have to be a pre-defined table.
Also, because the websocket replies would be different formats for different questions, I could not know in advance all the fields ever needed and even if so, most fields would not be used for differing replies.
My workable idea here is to create the reply tables using a field for the randomized key (which is still routed back round-trip through the websocket) and another large field to just store the JSON reply entirely.
Then in test_panel_function which is blocking in a loop waiting for the results, pull the JSON from the table, delete the row, and decode. Then the reply can be given back to the Javascript in the Django template.
3) Use Django signals
Django has a signals capability, but the response function doesn't seem to be able to be embedded (like inside test_panel_function) and there seems to be no wait() function available for an arbitrary function to just wait for the signal. If this were available, it would be very helpful

JavaScript, PHP - Using server-sent events with multiple, unique users?

I'm brand-new to SSE. There are plenty of simple/introductory texts on server sent events
Here
Here
Here
But none of them touch on using the same SSE file for multiple different users.
For example:
I have a site, where users log on. While logged on, they can view data that is unique and private to them. I'd like each user to have live updates while they are logged in, which may or may-not contain sensitive information. To do so I am implementing server-sent events:
JS (Straight out of one of the links)
var source;
if (!!window.EventSource) {
source = new EventSource("sse.php");
} else {
...
}
source.addEventListener("message", function(e) {
... do stuff ...
}, false);
PHP
header("Content-Type: text/event-stream");
header("Cache-Control: no-cache");
header("Connection: keep-alive");
while (true) {
$data = \\ query DB or any other source
if ($data) {
sendMessage($someID, $data);
}
sleep(2);
}
function sendMessage($id, $data) {
echo "id: $id\n";
echo "data: $data\n\n";
ob_flush();
flush();
}
But using it this way, am I only sending the data to the user who opened the sse.php source (i.e. each EventSource("sse.php") is a new, unique connection)? Or will everyone logged on, who has initialized the connection, receive the same data?
This SO answer touches on multiple users, by using unique events:
echo "event: ping\n";
$msg1="This is first user";
echo 'data: {"msg": "' . $msg1 . '"}';
echo "\n\n";
echo "event: notify\n";
$msg2="This is second user";
echo 'data: {"msg": "' . $msg2 . '"}';
echo "\n\n";
And then only listening for a certain event:
var evtSource = new EventSource("sender.php");
evtSource.addEventListener("ping", function(e) {
var obj = JSON.parse(e.data);
var r_msg = obj.msg;
But that hardly seems reasonable. If each user is supposed to be receiving their own private feed, then this method would require me to hard-code a unique event for each user. AND, as the original answer mentions, it does not prevent a slightly above-average user from intercepting and reading all the other messages too.
And, on top of it all, the sse.php file doesn't know which user it's gathering data for, so it would have to do all users, all the time.
How do I create one sse.php file, that can handle an unlimited number of unique users, and only send the data to the appropriate user? So:
there must be a way to send some initialization data (i.e. unique user ID) to server-side file, and
there must be a way for only that one particular user to receive the information gathered

am I only sending the data to the user who opened the sse.php source
Yes.
Or will everyone logged on, who has initialized the connection, receive the same data?
No.
If each user is supposed to be receiving their own private feed,
then this method would require me to hard-code a unique event for each user
Luckily no! The SSE "event" is for sending different types of events down a single connection to a single user. A use-case might be Facebook wanting to use one event for sending chat updates, another event for sending friend requests, another for sending ads to show, etc.
In my book (Data Push Apps with HTML5 SSE, http://shop.oreilly.com/product/0636920030928.do - apologies for the plug!) I argue that it is redundant, and better included as part of the json object you are pushing.
And, on top of it all, the sse.php file doesn't know which user ...
Cookies are sent. So the typical approach is to first login the user, create a cookie that authenticates them, then call the sse script. If using a server system with sessions support (e.g. PHP), then the cookies are an implementation detail.
POST data, and custom headers, cannot be sent. So if cookies are not an option you'd have to use GET to post some authentication id (but as you note, that is not the best type of security).

An SSE is an open HTTP connection. So it will only send unique data to a user if that is how you program it. You can make this stateful by using session, cookies, IP addresses, etc. in the same was as any other HTTP request. The main limitation of SSE is that you cannot add any headers to request to connect.

Accessing session in PHP command line?

I understand that command line is no web server, so you can't access $_SESSION. But I don't know what else to do.
I've been following this tutorial to create a chat using websockets: http://www.phpbuilder.com/articles/application-architecture/optimization/creating-real-time-applications-with-php-and-websockets.html
My problem is that, I do not know how to get the username of the message sender securely. I could include it in the message send function but since it's in Javascript, everyone can just edit their username to someone elses.
How could I securely get the username of the user, which is $_SESSION['username']?
var Server;
Server = new FancyWebSocket('ws://0.0.0.0:9000');
send( "test" );
I'm open to all kind of suggestions, like alternatives to websockets. I'm creating a realtime chat for my website.

First alternative is, of course, AJAX requests. AJAX doesn't have the problems of not being able to quickly and easily access the sessions that WebSockets has. Any sufficiently frequent sample rate is indistinguishable from real time.
Now, to my rather long-winded solution implemented in WebSockets:
The HTTP headers are available to the WebSocket server during the handshake, including the cookies. In the server that you're using, PHP-Websockets, the headers are stored in the $headers property.
For instance:
var_dump($user->headers);
array(14) {
["get"]=>
string(8) "/echobot"
["host"]=>
string(14) "127.0.0.1:9000"
...snip...
["cookie"]=>
string(100) "PHPSESSID=jan9uknpc06mk4ddghph4870t1; MyCookie=My+Value%21%40%23%24%25; MyNonhttponlyCookie=My+Value"
}
These cookies were generated from
session_start();
$_SESSION['Hi!'] = array('Hello!', 'where' => 'world!');
setcookie('MyCookie', 'My Value;!##$%', 0, '/', '127.0.0.1', false, true);
setcookie('MyNonhttponlyCookie', 'My Value', 0, '/', '127.0.0.1', false, false);
Thus, the value of $user->headers['cookie'] is a semicolon and space (;) delimited collection of key value pairs, where the values are URL encoded and separated from its key with an equal sign. (PHP complains if you put reserved characters in the cookie name. Thus the cookie key can not contain any url encoded values.)
A quick way to extract these are as follows
$cookies = array();
$cookiesParts = explode('; ', $user->headers['cookie']);
foreach ($cookiesParts as $cookieParts) {
$interimCookie = explode('=', $cookieParts);
$cookies[$interimCookie[0]] = urldecode($interimCookie[1]);
}
var_dump($cookies);
array(3) {
["PHPSESSID"]=>
string(26) "jan9uknpc06mk4ddghph4870t1"
["MyCookie"]=>
string(14) "My Value;!##$%"
["MyNonhttponlyCookie"]=>
string(8) "My Value"
}
We now have the session ID. Double check with session_name(), which will give you the key of the cookie that actually holds the session ID.
We could serialize and unserialize the session file as stored in the server, which is pointed at by session_save_path()... but I want to cheat.
Because the built-in session system locks the session files, we can't just keep the session file open and constantly watch for changes, nor can we lock the file ourselves for long periods of time.
It would be ideal if we could use the __get() and __set() magic methods here in the same way we'd use the $_SESSION superglobal (such as $myUser->_session['key'] = 'value';), but PHP does not allow treating these methods as arrays. Instead, we have to set a more mundanely named method.
<?php
class MyUser extends WebSocketUser {
public $session_id; // gets set somewhere. Good place is probably is your implementation of the `connected($user)` abstract method.
public getSession($key) {
session_id($this->session_id);
session_start();
$val = $_SESSION[$key];
session_write_close(); // very important!
return $val;
}
public setSession($key, $value) {
session_id($this->session_id);
session_start();
$_SESSION[$key] = value;
session_write_close(); // still very important!
}
}
(Note: I'm also pointing my feature request at this question, to base my eventual implementation of cookie parsing and session handling here, so that I can remember my research tonight as I work.)

Obfuscating URL parameters to prevent users from changing them?

I am developing a fat client page based on Javascript that will allow users to carry out tasks outwith another web client application (Oracle Siebel).
The way the web page will be called from the browser will be by a simple window.open() call.
When this happens a URL will be passed which contains some parameters at the end which will change the functionality of the fat client page depending on what value they have.
e.g
userlevel=1 //normal user
userlevel=2 //advanced user
In an example full URL would be like so
www.mypage.com/index.htm?id=25215125%userlevel=2%context=full
However a user who wants to change their access only need to figure out that if they change their user level then they can change their access rights on this fat client page.
Yes, I know this is risky and before you ask why I am not using a server supported thin client with controls that cannot be altered by the user. I simply have to do it this way!
This system will be in a "trusted" environment and this users will have at best average IT skills.
So all I need to do is figure out a way to obfuscate/ scramble the URL parameters (if possible) and then decipher them at the fat client.
e.g.
www.mypage.com/index.htm?1sdf908ga90-821098650f8asdg098g0a98
I tested it out on the browser and no complaints so far so I guess I just need to develop a piece of logic to decipher it.
e.g. I could use MD5?
Any examples or ideas?
Thanks

Try Base64 encoding it. https://stackoverflow.com/a/4699739/1088652
That'll shorten it and obfuscate it, so that users can't just throw values in the URL.

Params integrity can be ensured with HMAC. You generate hash using secret key and all the params, you include this hash inside of URL, then at server side you generate hash using same params and compare values.
function generateSignature(array $params, $hmacKey)
{
// sort the array by key using SORT_STRING order
ksort($params, SORT_STRING);
$escapeFunc = function ($val) {
return str_replace(':', '\\:', str_replace('\\', '\\\\', $val));
};
// generate the signing data string
$signData = implode(':', array_map($escapeFunc, array_merge(array_keys($params), array_values($params))));
// base64-encode the binary result of the HMAC computation
$merchantSig = base64_encode(hash_hmac('sha256', $signData, pack("H*", $hmacKey), true));
return $merchantSig;
}

Using XMLHTTPRequest to extract data from Database

I want to extract some data from the database without refreshing a page. What is the best possible way to do this?
I am using the following XMLHTTPRequest function to get some data (shopping cart items) from cart.php file. This file performs various functions based on the option value.
For example: option=1 means get all the shopping cart items. option=2 means delete all shopping cart items and return string "Your shopping cart is empty.". option=3, 4...and so on.
My XHR function:
function getAllCartItems()
{
if(window.XMLHttpRequest)
{
allCartItems = new XMLHttpRequest();
}
else
{
allCartItems=new ActiveXObject("Microsoft.XMLHTTP");
}
allCartItems.onreadystatechange=function()
{
if (allCartItems.readyState==4 && allCartItems.status==200)
{
document.getElementById("cartmain").innerHTML=allCartItems.responseText;
}
else if(allCartItems.readyState < 4)
{
//do nothing
}
}
var linktoexecute = "cart.php?option=1";
allCartItems.open("GET",linktoexecute,true);
allCartItems.send();
}
cart.php file looks like:
$link = mysql_connect('localhost', 'user', '123456');
if (!$link)
{
die('Could not connect: ' . mysql_error());
}
mysql_select_db('projectdatabase');
if($option == 1) //get all cart items
{
$sql = "select itemid from cart where cartid=".$_COOKIE['cart'].";";
$result = mysql_query($sql);
$num = mysql_num_rows($result);
while($row = mysql_fetch_array($result))
{
echo $row['itemid'];
}
}
else if($option == 2)
{
//do something
}
else if($option == 3)
{
//do something
}
else if($option == 4)
{
//do something
}
My Questions:
Is there any other way I can get the data from database without
refreshing the page?
Are there any potential threats (hacking, server utilization,
performance etc) in my way of doing this thing? I believe a hacker
can flood my server be sending unnecessary requests using option=1,
2, 3 etc.

I don't think a Denial of Service attack would be your main concern, here. That concern would be just as valid is cart.php were to return HTML. No, exposing a public API for use via AJAX is pretty common practice.
One thing to keep in mind, though, is the ambiguity of both listing and deleting items via the same URL. It would be a good idea to (at the very least) separate those actions (or "methods") into distinct URLs (for example: /cart/list and /cart/clear).
If you're willing to go a step further, you should consider implementing a "RESTful" API. This would mean, among other things, that methods can only be called using the correct HTTP verb. You've possibly only heard of GET and POST, but there's also PUT and DELETE, amongst others. The reason behind this is to make the methods idempotent, meaning that they do the same thing again and again, no matter how many times you call them. For example, a GET call to /cart will always list the contents and a DELETE call to /cart will always delete all items in the cart.
Although it is probably not practical to write a full REST API for your shopping cart, I'm sure some of the principles may help you build a more robust system.
Some reading material: A Brief Introduction to REST.

Ajax is the best option for the purpose.
Now sending and receiving data with Ajax is done best using XML. So use of Web services is the recommended option from me. You can use a SOAP / REST web service to bring data from a database on request.
You can use this Link to understand more on Webservices.
For the tutorials enough articles are available in the Internet.

you're using a XMLHttpRequest object, so you don't refresh your page (it's AJAX), or there's something you haven't tell
if a hacker want to DDOS your website, or your database, he can use any of its page... As long as you don't transfer string between client and server that will be used in your SQL requests, that should be OK
I'd warn you about the use of raw text response display. I encourage you to format your response as XML or JSON to correctly locate objects that needs to be inserted into the DOM, and to return an tag to correctly handle errors (the die("i'm your father luke") won't help any of your user) and to display them in a special area of your web page

First, you should consider separating different parts of your application. Having a general file that performs every other tasks related to carts, violates all sorts of software design principles.
Second, the first vulnerability is SQL injection. You should NEVER just concatenate the input to your SQL.
Suppose I posted 1; TRUNCATE TABLE cart;. Then your SQL would look like:
select itemid from cart where cartid=1; TRUNCATE TABLE cart; which first selects the item in question, then ruins your database.
You should write something like this:
$item = $_COOKIE['cart'];
$item = preg_replace_all("['\"]", "\\$1", $item);
To avoid refreshing, you can put a link on your page. Something like, Refresh
In terms of security, it will always pay to introduce a database layer concerned with just your data, regardless of your business logic, then adding a service layer dependent on the database layer, which would provide facilities to perform business layer actions.
You should also take #PPvG recommendation into note, and -- using Apache's mod_rewrite or other similar facilities -- make your URLs more meaningful.
Another note: try to encapsulate your data in JSON or XML format. I'd recommend the use of json_encode(); on the server side, and JSON.parse(); on the client side. This would ensure a secure delivery.