Develop Reference

webscraping logins with javscript and ajax

i'm stuck trying to get viewstate data to pass for form authentication. heres my code:
import scrapy
from scrapy_splash import SplashRequest, SplashFormRequest
class BrpSplashSpider(scrapy.Spider):
name = 'brp_splash'
allowed_domains = ['brp.secure.force.com']
# start_urls = ['http://brp.secure.force.com/']
script = '''
function main(splash, args)
url = args.url
assert(splash:go(url))
assert(splash:wait(1))
return splash:html()
end
'''
def start_requests(self):
yield SplashRequest(
url='https://brp.secure.force.com/login',
endpoint = 'execute',
args = {
'lua_source':self.script
},
callback=self.parse
)
def parse(self,response):
yield SplashFormRequest.from_response(
response,
formxpath='//form',
formdata={
'AJAXREQUEST' : '_viewRoot'
'j_id0:j_id5' : 'j_id0:j_id5'
'j_id0:j_id5:Login1_Dealer_No': '******'
'j_id0:j_id5:Login1_UserName': '******'
'j_id0:j_id5:Login1_Password': '*********'
'j_id0:j_id5:j_id26': 'en'
'com.salesforce.visualforce.ViewState':
}
)
pass
inspecting the webpage and looking at the form data i can see a huge string that is the viewstate data. its not in the html. where is it and how do i reference it?
thanks for looking,
jim

the object i'm looking for is in the login page. i was looking for it after logging in. newbie learning curve mistake

Why an event handler function in consumers.py is called multiple times and then web-socket connection is killed forcefully?

I have created a real time notifier using web-socket, implemented using Django-Channels. It sends the "Post Author" a real time notification whenever any user likes the author's post.
The problem is, for single like button click, multiple notifications are being saved in database. This is because the function (like_notif()) defined in consumers.py which is saving notification data to database when the 'like button click' event happens is called multiple times (when it should be called only once) until the web-socket is disconnected.
In the following lines I will provide detail of what is actually happening(to the best of my understanding) behind the scene.
Like button is clicked.
<a class="like-btn" id="like-btn-{{ post.pk }}" data-likes="{{ post.likes.count }}" href="{% url 'like_toggle' post.slug %}">
Like
</a>
JavaScript prevents the default action and generates an AJAX call to a URL.
$('.like-btn').click(function (event) {
event.preventDefault();
var this_ = $(this);
var url = this_.attr('href');
var likesCount = parseInt(this_.attr('data-likes')) || 0;
$.ajax({
url: url,
method: "GET",
data: {},
success: function (json) {
// DOM is manipulated accordingly.
},
error: function (json) {
// Error.
}
});
});
The URL is mapped to a function defined in views.py. (Note: Code for Post Model is at the last if required for reference.)
# In urls.py
urlpatterns = [
path('<slug:slug>/like/', views.post_like_toggle, name="like_toggle"),
]
# In views.py
#login_required
def post_like_toggle(request, slug):
"""
Function to like/unlike posts using AJAX.
"""
post = Post.objects.get(slug=slug)
user = request.user
if user in post.likes.all():
# Like removed
post.likes.remove(user)
else:
# Like Added
post.likes.add(user)
user_profile = get_object_or_404(UserProfile, user=user)
# If Post author is not same as user liking the post
if str(user_profile.user) != str(post.author):
# Sending notification to post author is the post is liked.
channel_layer = get_channel_layer()
text_dict = {
# Contains some data in JSON format to be sent.
}
async_to_sync(channel_layer.group_send)(
"like_notif", {
"type": "notif_like",
"text": json.dumps(text_dict),
}
)
response_data = {
# Data sent to AJAX success/error function.
}
return JsonResponse(response_data)
Consumer created to handle this in consumers.py.
# In consumers.py
class LikeNotificationConsumer(AsyncConsumer):
async def websocket_connect(self, event):
print("connect", event)
await self.channel_layer.group_add("like_notif", self.channel_name)
await self.send({
"type": "websocket.accept",
})
async def websocket_receive(self, event):
print("receive", event)
async def websocket_disconnect(self, event):
print("disconnect", event)
await self.channel_layer.group_discard("like_notif", self.channel_name)
async def notif_like(self, event):
print("like_notif_send", event)
await self.send({
"type": "websocket.send",
"text": event.get('text')
})
json_dict = json.loads(event.get('text'))
recipient_username = json_dict.get('recipient_username')
recipient = await self.get_user(recipient_username)
sender_username = json_dict.get('sender_username')
sender = await self.get_user(sender_username)
post_pk = json_dict.get('post_pk', None)
post = await self.get_post(post_pk)
verb = json_dict.get('verb')
description = json_dict.get('description', None)
data_dict = json_dict.get('data', None)
data = json.dumps(data_dict)
await self.create_notif(recipient, sender, verb, post, description, data)
#database_sync_to_async
def get_user(self, username_):
return User.objects.get(username=username_)
#database_sync_to_async
def get_post(self, pk_):
return Post.objects.get(pk=pk_)
#database_sync_to_async
def create_notif(self, recipient, sender, verb, post=None, description=None, data=None, *args, **kwargs):
return Notification.objects.create(recipient=recipient, sender=sender, post=post, verb=verb, description=description, data=data)
Note: notif_like() function is being called multiple times and thus data is being saved multiple times in the database. Why is this function being called multiple times?
JavaScript code(written in HTML file) handling websocket connection. I have used Reconnecting Websockets .
<script type="text/javascript">
var loc = window.location
var wsStart = 'ws://'
if(loc.protocol == 'https:') {
wsStart = 'wss://'
}
var endpoint = wsStart + loc.host + "/like_notification/"
var likeSocket = new ReconnectingWebSocket(endpoint)
likeSocket.onmessage = function (e) {
console.log("message LikeNotificationConsumer", e)
var data_dict = JSON.parse(e.data)
/*
DOM manipulation using data from data_dict.
NOTE: All the intended data is coming through perfectly fine.
*/
};
likeSocket.onopen = function (e) {
console.log("opened LikeNotificationConsumer", e)
}
likeSocket.onerror = function (e) {
console.log("error LikeNotificationConsumer", e)
}
likeSocket.onclose = function (e) {
console.log("close LikeNotificationConsumer", e)
}
</script>
Post model for reference
# In models.py
class Post(models.Model):
title = models.CharField(max_length=255, blank=False, null=True, verbose_name='Post Title')
post_content = RichTextUploadingField(blank=False, null=True, verbose_name='Post Content')
author = models.ForeignKey(
User,
on_delete=models.CASCADE,
default=None,
blank=True,
null=True
)
likes = models.ManyToManyField(User, blank=True, related_name='post_likes')
published = models.DateTimeField(default=datetime.now, blank=True)
tags = models.ManyToManyField(Tags, verbose_name='Post Tags', blank=True, default=None)
slug = models.SlugField(default='', blank=True)
def save(self, *args, **kwargs):
self.slug = slugify(self.title + str(self.pk))
super(Post, self).save()
def get_like_url(self):
return reverse("like_toggle", (), {'slug', self.slug})
def __str__(self):
return '%s' % self.title
Notification model for reference
# In models.py
class Notification(models.Model):
recipient = models.ForeignKey(User, blank=False, on_delete=models.CASCADE, related_name="Recipient",)
sender = models.ForeignKey(
User,
blank=False,
on_delete=models.CASCADE,
related_name="Sender"
)
# Post (If notification is attached to some post)
post = models.ForeignKey(
Post,
blank=True,
default=None,
on_delete=models.CASCADE,
related_name="Post",
)
verb = models.CharField(max_length=255)
description = models.TextField(blank=True, null=True)
data = models.TextField(blank=True, null=True)
timestamp = models.DateTimeField(default=timezone.now)
unread = models.BooleanField(default=True, blank=False, db_index=True)
def __str__(self):
return self.verb
Channels layer setting.
# In settings.py
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {
"hosts": [("localhost", 6379)]
}
}
}
I expect the notification to be saved only once per click on the like button but it is being saved multiple times as the notif_like() function defined in consumers.py is being called multiple times per click.
Moreover, web-socket gets disconnected abruptly after this and following warning is displayed in the terminal -
Application instance <Task pending coro=<SessionMiddlewareInstance.__call__() running at /home/pirateksh/DjangoProjects/ccwebsite/ccenv/lib/python3.7/site-packages/channels/sessions.py:183> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fb7603af1c8>()]>> for connection <WebSocketProtocol client=['127.0.0.1', 52114] path=b'/like_notification/'> took too long to shut down and was killed.
I am struggling with this for quite a time now, so it will be really helpful if someone can guide me through this.

After almost a year I was working on a very similar project and I was able to reproduce this bug and also figured out what was causing this.
I am pretty sure that this is somehow being caused by ReconnectingWebSocket.
The reason for this is that when I switched back to using vanilla WebSocket, everything worked fine.
I have also opened an issue regarding this on the ReconnectingWebSocket's Github repository.

How can I seperate data by sessions/requests

I am doing a song search engine as a project and everything works fine except when I run the program from 2 different computers the data sent to each computer is mixed with the data sent to the other computer
I've tried separating the users with variables in sessions but the variable is much bigger than the max value a session variable can hold.
I've tried also making a different URL based on the request but for some reason it sends a 404 error.
This is the python code relevant
#app.route("/", methods=['GET','POST'])
def index():
global str_request
global search_done
global songs
global already_requested
already_requested=False
api_connection=connected_to_internet()
print(api_connection)
if api_connection:
if request.method == 'POST':
search_request = request.form.getlist('search[]')
str_request=turn_list_to_str(search_request)
str_request=str_request.replace(' ','%20')
print ("/result/"+str_request)
search_request=arrange_request(search_request)
print(search_request)
songs=[]
data=search_api(search_request)
search_done=False
if data!=None:
for hit in data:
songs.append(Song(hit['result']['full_title'],hit['result']['primary_artist']['name'],hit['result']['url'],hit['result']['song_art_image_thumbnail_url']))
return render_template('layout.html')
else:
return render_template('connection.html')
#app.route("/result/"+str_request)
def result():
global songs
return render_template("result.html",songs=songs)
this is the js code relevant
$(document).ready(function(){
$("#search_button").click(function(){
btn=document.getElementById("search_button");
btn.disabled=true;
var $loader = $("<div />");
$loader.addClass('loader');
$loader.appendTo("body");
var search_result=arrangeTag();
//post https://lyritag.xyz/
$.post("http://localhost/",
{
"search":search_result
},function(){
//var user_request = '<%=Session["user_request"]%>';
//alert(user_request);
for(var i = 0; i < search_result.length; i++){
console.log(search_result[i])
}
location.replace("/result/"+search_result);
})
});
});
Both the URL and the app.route of the result are the same but It returns a 404 error.
Any other solution will be great as well.

How do I send data from JS to Python with Flask?

I'm making a website with Flask and I'd like to be able to execute python code using data from the page. I know that I can simply use forms but it's a single page that is continually updated as it receives user input and it'd be a massive pain in the ass to have it reload the page every time something happens. I know I can do {{ function() }} inside the javascript but how do I do {{ function(args) }} inside the javascript using js variables? So far the only thing I can think of is to update an external database like MongoDB with the js then use Python to read from that, but this process will slow down the website quite a lot.
The jQuery needs to get a list of dictionary objects from the Python function which can then be used in the html. So I need to be able to do something like:
JS:
var dictlist = { getDictList(args) };
dictlist.each(function() {
$("<.Class>").text($(this)['Value']).appendTo("#element");
});
Python:
def getDictList(args):
return dictlistMadeFromArgs

To get data from Javascript to Python with Flask, you either make an AJAX POST request or AJAX GET request with your data.
Flask has six HTTP methods available, of which we only need the GET and POST. Both will take jsdata as a parameter, but get it in different ways. That's how two completely different languages in two different environments like Python and Javascript exchange data.
First, instantiate a GET route in Flask:
#app.route('/getmethod/<jsdata>')
def get_javascript_data(jsdata):
return jsdata
or a POST one:
#app.route('/postmethod', methods = ['POST'])
def get_post_javascript_data():
jsdata = request.form['javascript_data']
return jsdata
The first one is accessed by /getmethod/<javascript_data> with an AJAX GET as follows:
$.get( "/getmethod/<javascript_data>" );
The second one by using an AJAX POST request:
$.post( "/postmethod", {
javascript_data: data
});
Where javascript_data is either a JSON dict or a simple value.
In case you choose JSON, make sure you convert it to a dict in Python:
json.loads(jsdata)[0]
Eg.
GET:
#app.route('/getmethod/<jsdata>')
def get_javascript_data(jsdata):
return json.loads(jsdata)[0]
POST:
#app.route('/postmethod', methods = ['POST'])
def get_post_javascript_data():
jsdata = request.form['javascript_data']
return json.loads(jsdata)[0]
If you need to do it the other way around, pushing Python data down to Javascript, create a simple GET route without parameters that returns a JSON encoded dict:
#app.route('/getpythondata')
def get_python_data():
return json.dumps(pythondata)
Retrieve it from JQuery and decode it:
$.get("/getpythondata", function(data) {
console.log($.parseJSON(data))
})
The [0] in json.loads(jsdata)[0] is there because when you decode a JSON encoded dict in Python, you get a list with the single dict inside, stored at index 0, so your JSON decoded data looks like this:
[{'foo':'bar','baz':'jazz'}] #[0: {'foo':'bar','baz':'jazz'}]
Since what we need is the just the dict inside and not the list, we get the item stored at index 0 which is the dict.
Also, import json.

.html
... id="clickMe" onclick="doFunction();">
.js
function doFunction()
{
const name = document.getElementById("name_").innerHTML
$.ajax({
url: '{{ url_for('view.path') }}',
type: 'POST',
data: {
name: name
},
success: function (response) {
},
error: function (response) {
}
});
};
.py
#app.route("path", methods=['GET', 'POST'])
def view():
name = request.form.get('name')
...

im new in coding, but you can try this:
index.html
<script>
var w = window.innerWidth;
var h = window.innerHeight;
document.getElementById("width").value = w;
document.getElementById("height").value = h;
</script>
<html>
<head>
<!---Your Head--->
</head>
<body>
<form method = "POST" action = "/data">
<input type = "text" id = "InputType" name = "Text">
<input type = "hidden" id = "width" name = "Width">
<input type = "hidden" id = "height" name = "Height">
<input type = "button" onclick = "myFunction()">
</form>
</body>
</html>
.py
from flask import Flask, request
app = Flask(__name__)
html = open("index.html").read()
#app.route("/")
def hello():
return html
#app.route("/data", methods=["POST", "GET"])
def data():
if request.method == "GET":
return "The URL /data is accessed directly. Try going to '/form' to submit form"
if request.method == "POST":
text = request.form["Text"]
w = request.form["Width"]
h = request.form["Height"]
//process your code
return //value of your code

Django: SyntaxError: Unexpected token <

I'm using django with jquery to upload files using ajax. However, after the file has uploaded (and the files do upload correctly; the database is updated and they are on the server) javascript returns with an SyntaxError: Unexpected token < error.
I don't know if this is an error in javascript or an error in django; other questions I saw seemed to recommend making sure the information is indeed in JSON format, but it seems to be properly formatted in my case. I don't really understand why there is an error, since everything seems to work fine. There are no errors in my logs.
If I could only see a more helpful error message, I would have a much easier time fixing the issue.
Here's my views.py:
def response_mimetype(request):
if "application/json" in request.META['HTTP_ACCEPT']:
return "application/json"
else:
return "text/plain"
class JSONResponse(HttpResponse):
"""JSON response class."""
def __init__(self,obj='',json_opts={},mimetype="application/json",*args,**kwargs):
content = simplejson.dumps(obj,**json_opts)
a = super(JSONResponse,self).__init__(content,mimetype,*args,**kwargs)
class UploadedFileCreateView(CreateView):
model = UploadedFile
form_class = UploadedFileForm
def form_valid(self, form):
self.object = form.save(commit=False)
self.object.project_id = self.kwargs['proj_key']
self.object.save()
f = self.request.FILES.get('file')
data = [{
'name': self.object.name(),
'url': "/uploads/xmlfiles/" + self.object.name().replace(" ", "_"),
'delete_url': reverse('fileupload:upload-delete',
kwargs={'pk':self.object.id,
'proj_key':self.kwargs['proj_key']}),
'delete_type': "DELETE"}]
response = JSONResponse(data, {}, response_mimetype(self.request))
response['Content-Disposition'] = 'inline; filename=files.json'
return super(UploadedFileCreateView, self).form_valid(form)
def get_context_data(self, **kwargs):
context = super(UploadedFileCreateView, self).get_context_data(**kwargs)
context['files'] = UploadedFile.objects.all()
context['proj'] = int(self.kwargs["proj_key"])
return context
And here's my models.py:
class UploadedFile(models.Model):
"""This represents a file that has been uploaded to the server."""
STATE_UPLOADED = 0
STATE_ANNOTATED = 1
STATE_PROCESSING = 2
STATE_PROCESSED = 4
STATES = (
(STATE_UPLOADED, "Uploaded"),
(STATE_ANNOTATED, "Annotated"),
(STATE_PROCESSING, "Processing"),
(STATE_PROCESSED, "Processed"),
)
status = models.SmallIntegerField(choices=STATES,
default=0, blank=True, null=True)
file = models.FileField(upload_to=settings.XML_ROOT)
project = models.ForeignKey(Project)
def __unicode__(self):
return self.file.name
def name(self):
return os.path.basename(self.file.name)
def save(self, *args, **kwargs):
if not self.status:
self.status = self.STATE_UPLOADED
super(UploadedFile, self).save(*args, **kwargs)
def delete(self, *args, **kwargs):
os.remove(self.file.path)
self.file.delete(False)
super(UploadedFile, self).delete(*args, **kwargs)
def get_absolute_url(self):
return u'/upload/projects/%d' % self.id
class UploadedFileForm(ModelForm):
class Meta:
model = UploadedFile
exclude = ('project',)

According to the documentation here, the form_valid() method needs to return the JSON as a HTTPResponse. Calling the form_valid() on the super class will, instead, overwrite the response created on the previous two lines.
Edit - This is working for me:
def response_mimetype(request):
if "application/json" in request.META['HTTP_ACCEPT']:
return "application/json"
else:
return "text/plain"
class UploadedFileCreateView(CreateView):
model = UploadedFile
form_class = UploadedFileForm
def form_valid(self, form):
self.object = form.save(commit=False)
self.object.project_id = self.kwargs['proj_key']
self.object.save()
f = self.request.FILES.get('file')
data = [{
'name': self.object.name(),
'url': "/uploads/xmlfiles/" + self.object.name().replace(" ", "_"),
'type': "application/xml",
'size': self.object.file.size,
'delete_url': reverse('fileupload:upload-delete',
kwargs={'pk':self.object.id,
'proj_key':self.kwargs['proj_key']}),
'delete_type': "DELETE"}]
return HttpResponse(simplejson.dumps(data), content_type = response_mimetype(self.request))
The code in the link above bundles the ajax response code as a Mixin which would allow other forms in the project to reuse the code.
Viewing the response to the Ajax call in the Firefox plugin Firebug, Net tab or Chrome equivalent will help. These will show both the request and response headers ensuring the correct settings are being sent and returned.