webscraping logins with javscript and ajax - javascript

i'm stuck trying to get viewstate data to pass for form authentication. heres my code:
import scrapy
from scrapy_splash import SplashRequest, SplashFormRequest
class BrpSplashSpider(scrapy.Spider):
name = 'brp_splash'
allowed_domains = ['brp.secure.force.com']
# start_urls = ['http://brp.secure.force.com/']
script = '''
function main(splash, args)
url = args.url
assert(splash:go(url))
assert(splash:wait(1))
return splash:html()
end
'''
def start_requests(self):
yield SplashRequest(
url='https://brp.secure.force.com/login',
endpoint = 'execute',
args = {
'lua_source':self.script
},
callback=self.parse
)
def parse(self,response):
yield SplashFormRequest.from_response(
response,
formxpath='//form',
formdata={
'AJAXREQUEST' : '_viewRoot'
'j_id0:j_id5' : 'j_id0:j_id5'
'j_id0:j_id5:Login1_Dealer_No': '******'
'j_id0:j_id5:Login1_UserName': '******'
'j_id0:j_id5:Login1_Password': '*********'
'j_id0:j_id5:j_id26': 'en'
'com.salesforce.visualforce.ViewState':
}
)
pass
inspecting the webpage and looking at the form data i can see a huge string that is the viewstate data. its not in the html. where is it and how do i reference it?
thanks for looking,
jim

the object i'm looking for is in the login page. i was looking for it after logging in. newbie learning curve mistake

Related

Django REST Framework - "Must be Device instance" error when calling PUT from javascript

I'm getting an error when trying to update an object instance using PUT using the rest framework:
ValueError: Cannot assign "{'id': UUID('954...8b4')}": "DeviceConfig.device" must be a "Device" instance.
view defn:
class DeviceConfigViewSet(viewsets.ModelViewSet):
#todo: secure
authentication_classes = []
queryset = DeviceConfig.objects.all().order_by('device')
def get_queryset(self):
device = self.get_renderer_context()["request"].query_params.get('device')
if device:
return DeviceConfig.objects.filter(device=Device(device=device))[0:]
else:
return self.queryset
serializer_class = DeviceConfigSerializer
DeviceConfig model:
class DeviceConfig(models.Model):
device = models.OneToOneField(Device,primary_key=True,on_delete=models.CASCADE)
bpm = models.DecimalField(max_digits=22, decimal_places=3, blank=True, null=True, default=0.5)
duty = models.DecimalField(max_digits=22, decimal_places=4, blank=True, null=True, default=0.022)
ledState = models.IntegerField(
default=255,
validators=[MaxValueValidator(255), MinValueValidator(0)]
)
pressureMax = models.IntegerField( blank=True, null=True, default=70, validators=[MaxValueValidator(255), MinValueValidator(10)])
JS func FE side:
function formSubmit(){
var myForm = document.getElementById("config-form");
var formObj = new FormData(myForm);
var object = {};
formObj.forEach(function(value, key){
object[key] = value;
});
object["device"] = DEVICE_UUID;
const putMethod = {
method: 'PUT',
headers: {
'Content-type': 'application/json; charset=UTF-8' // Indicates the content
},
body: JSON.stringify(object)
}
fetch("http://localhost:81/configs/"+DEVICE_UUID, putMethod);
}
I've tried not sending the device ID from the front end but it gives me a 400 then
the ol' goog search didn't turn up much for me this time, but I'm not quite sure how to send a Device instance from the client side besides by it's pk
which is what I've tried
EDIT - DeviceConfigSerializer
class DeviceConfigSerializer(serializers.HyperlinkedModelSerializer):
device = serializers.UUIDField(source='device.id')
class Meta:
model = DeviceConfig
fields = ('device','bpm', 'duty','ledState','pressureMax')
I worked it out. #JPG helped point in the right direction, it was the serializer - i was needlessly referencing device, which was screwing things up. I've removed it from the serializer and JS and all works well now. Thanks for pointing me in the right direction #JPG
Given the reference to the device ID was already in the url I was calling, there's no need to pass it in again. I'm still not quite sure how to interpret the error (insight appreciated) but my thoughts are it was attempting to assign a plain UUID where it needed a Device instance, which was not being handled in my serializer code as I thought it was.
Cheers

How to pass python string to javascript as string in Django?

I have a python string which is javascript code and I want to pass this string to javascript as string too.
My idea is to pass python string to javascript string and then use eval() function in javascript to turn that string to actual code and execute it.
def login(request):
success = '''
window.location = '{% url 'home' %}';
# some other codes
'''
return render(request, "app/login.html", {'success': success})
var code = "{{success}}"
console.log(code) // return Uncaught SyntaxError: Invalid or unexpected token
I have also tried pass the string as json like this
def login(request):
success = '''
window.location = '{% url 'home' %}';
# some other codes
'''
success = json.dumps(success)
return render(request, "app/login.html", {'success': success})
var code = JSON.parse("{{success|safe}}");
console.log(code) //return Uncaught SyntaxError: missing ) after argument list
Last thing that I have tried is
def login(request):
success = '''
window.location = '{% url 'home' %}';
# some other codes
'''
return render(request, "app/login.html", {'success': success})
<h3 id="success_id" hidden>{{success}}</h3>
<script>
var code = $("#success_id").text();
console.log(code) //return window.location = "{% url 'home' %}"
// if i do this
var code = "window.location = '{% url 'home' %}'";
console.log(code) // return window.location = /app/home/
// I want it to return the url like /app/home
</script>
How can I do this?
Ok in essence what you want to do is using Django to pass data to js?
That has to to do with Django creating a Json Response which the JavaScript will then make a request , in essence you are working with some API.
your view code where you wrote the render function won't need that anymore...
Something like
JsonResponse({'success':success})
will be needed , after importing the JsonResponse from 'django.http'
Then at the js file, you will have to use Ajax to call that url with it's required method to get the data needed.
in your script tag , you will then need something like this
const xhr= new XMLHttpRequest ()
xhr.open('django_url', 'method')
xhr.onload = function () {
const msg = xhr.response.success
}
xhr.send()
If you are familiar with fetch or axios you can as well use any library of your choice.
the django_url refers to the url that connects that view
the method refers to the http_method, either a 'GET' or 'POST' (most commonly used) .

How to send JSON data created by Python to JavaScript?

I am using Python cherrypy and Jinja to serve my web pages. I have two Python files: Main.py (handle web pages) and search.py (server-side functions).
I create a dynamic dropdown list (using JavaScript) based on a local JSON file called component.json(created by function componentSelectBar inside search.py).
I want to ask how can my JavaScript retrieve JSON data without physically storing the JSON data into my local website root's folder and still fulfil the function of dynamic dropdown list.
The componentSelectBar function inside search.py:
def componentSelectBar(self, brand, category):
args = [brand, category]
self.myCursor.callproc('findComponent', args)
for result in self.myCursor.stored_results():
component = result.fetchall()
if (len(component) == 0):
print "component not found"
return "no"
components = []
for com in component:
t = unicodedata.normalize('NFKD', com[0]).encode('ascii', 'ignore')
components.append(t)
j = json.dumps(components)
rowarraysFile = 'public/json/component.json'
f = open(rowarraysFile, 'w')
print >> f, j
print "finish component bar"
return "ok"
The selectBar.js:
$.getJSON("static/json/component.json", function (result) {
console.log("retrieve component list");
console.log("where am i");
$.each(result, function (i, word) {
$("#component").append("<option>"+word+"</option>");
});
});
store results from componentSelectBar into database
expose new api to get results from database and return json to browser
demo here:
#cherrypy.expose
def codeSearch(self, modelNumber, category, brand):
...
result = self.search.componentSelectBar(cherrypy.session['brand'], cherrypy.session['category'])
# here store result into a database, for example, brand_category_search_result
...
#cherrypy.expose
#cherrypy.tools.json_out()
def getSearchResult(self, category, brand):
# load json from that database, here is brand_category_search_result
a_json = loadSearchResult(category, brand)
return a_json
document on CherryPy, hope helps:
Encoding response
In your broswer, you need to GET /getSearchResult for json:
$.getJSON("/getSearchResult/<arguments here>", function (result) {
console.log("retrieve component list");
console.log("where am i");
$.each(result, function (i, word) {
$("#component").append("<option>"+word+"</option>");
});
});
To use that json data directly into javascript you can use
var response = JSON.parse(component);
console.log(component); //prints
OR
You already created json file.If that file is in right format then you can read json data from that file using jQuery jQuery.getJSON() For more: http://api.jquery.com/jQuery.getJSON/
You are rendering a HTML and sending it as response. If you wish to do with JSON, this has to change. You should return JSON in your main.py, whereas you will send a HTML(GET or POST) from Javascript and render it back.
def componentSelectBar(self, brand, category):
/* Your code goes here */
j = json.dumps(components)
// Code to add a persistent store here
rowarraysFile = 'public/json/component.json'
f = open(rowarraysFile, 'w')
print >> f, j
// Better to use append mode and append the contents to the file in python
return j //Instead of string ok
#cherrypy.expose
def codeSearch(self):
json_request = cherrypy.request.body.read()
import json # This should go to the top of the file
input_dict = json.loads(json_request)
modelNumber = input_dict.get("modelNumber", "")
category = input_dict.get("category", "")
brand = input_dict.get("brand", "")
/* Your code goes here */
json_response = self.search.componentSelectBar(cherrypy.session['brand'], cherrypy.session['category'])
return json_response
Here, I added only for the successful scenario. However, you should manage the failure scenarios(a JSON error response that could give as much detail as possible) in the componentSelectBar function. That will help you keep the codeSearch function as plain as possible and help in a long run(read maintaining the code).
And I would suggest you to read PEP 8 and apply it to the code as it is kind of norm for all python programmers and help any one else who touches your code.
EDIT: This is a sample javascript function that will make a post request and get the JSON response:
searchResponse: function(){
$.ajax({
url: 'http://localhost:8080/codeSearch', // Add your URL here
data: {"brand" : "Levis", "category" : "pants"}
async: False,
success: function(search_response) {
response_json = JSON.parse(search_response)
alert(response_json)
// Do what you have to do here;
// In this specific case, you have to generate table or any structure based on the response received
}
})
}

Django + Angular method PUT and POST not allowed

I refactoring a project with Django/Django-Rest and AngularJS 1.4.9. All my GET requests are working fine, but PUT and POST requests don't. I receive a "405 (METHOD NOT ALLOWED)" error.
All my http requests were Ajax and worked fine, but I'm changing to $http now and having this trouble. What is wrong?
app.js
'use strict';
angular.module('myapp', [], function($interpolateProvider, $httpProvider) {
$interpolateProvider.startSymbol('{[{');
$interpolateProvider.endSymbol('}]}');
$httpProvider.defaults.xsrfCookieName = 'jxcsrf';
$httpProvider.defaults.xsrfHeaderName = 'X-CSRFToken';
}).run(function($http) {
$http.defaults.headers.common['Content-Type'] = "application/json";
});
UPDATE showing the django view (but I think is not a django problem, because worked with ajax)
The view
class MusicAction(APIView):
"""
Create a music instance or update the vote field in a music instance.
"""
permission_classes = (IsAllowedOrAdminOrReadOnly,)
def get_playlist(self, station, playlist_id):
try:
return Playlist.objects.get(station=station, pk=playlist_id, playing="1")
except Playlist.DoesNotExist:
raise ValidationError(u'Essa playlist não está em uso.')
def get_music(self, music_id, playlist_id):
try:
obj = Music.objects.get(music_id=music_id, playlist_id=playlist_id)
return obj
except Music.DoesNotExist:
raise ValidationError(u'Essa música já tocou ou não existe.')
def get_object(self, station_id):
try:
obj = Station.objects.get(pk=station_id, is_active=True)
self.check_object_permissions(self.request, obj)
return obj
except Station.DoesNotExist:
return Response(status=404)
def put(self, request, format=None):
station_id = request.data.get("sid")
station = self.get_object(station_id)
playlist_id = request.data.get("pid")
playlist = self.get_playlist(station, playlist_id)
music_id = request.data.get('mid')
music = self.get_music(music_id, playlist_id)
vote = int(request.data.get("vote"))
with transaction.atomic():
total = music.vote
music.vote = total + vote
music.save()
station.last_update = timezone.now()
station.save()
if vote > 0:
Voting.objects.create(voted_by=request.user, music=music)
else:
vote = Voting.objects.get(voted_by=request.user, music=music)
vote.delete()
serializer = MusicSerializer(music)
return Response(serializer.data)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
This is the header.
Try using one of the generic views of django rest framework:
http://www.django-rest-framework.org/api-guide/generic-views/
for example, if you'd like to accept get and put use the RetrieveUpdateAPIView:
from rest_framework import generics
class MusicAction(generics.RetrieveUpdateAPIView):
#code...

How do I send data from JS to Python with Flask?

I'm making a website with Flask and I'd like to be able to execute python code using data from the page. I know that I can simply use forms but it's a single page that is continually updated as it receives user input and it'd be a massive pain in the ass to have it reload the page every time something happens. I know I can do {{ function() }} inside the javascript but how do I do {{ function(args) }} inside the javascript using js variables? So far the only thing I can think of is to update an external database like MongoDB with the js then use Python to read from that, but this process will slow down the website quite a lot.
The jQuery needs to get a list of dictionary objects from the Python function which can then be used in the html. So I need to be able to do something like:
JS:
var dictlist = { getDictList(args) };
dictlist.each(function() {
$("<.Class>").text($(this)['Value']).appendTo("#element");
});
Python:
def getDictList(args):
return dictlistMadeFromArgs
To get data from Javascript to Python with Flask, you either make an AJAX POST request or AJAX GET request with your data.
Flask has six HTTP methods available, of which we only need the GET and POST. Both will take jsdata as a parameter, but get it in different ways. That's how two completely different languages in two different environments like Python and Javascript exchange data.
First, instantiate a GET route in Flask:
#app.route('/getmethod/<jsdata>')
def get_javascript_data(jsdata):
return jsdata
or a POST one:
#app.route('/postmethod', methods = ['POST'])
def get_post_javascript_data():
jsdata = request.form['javascript_data']
return jsdata
The first one is accessed by /getmethod/<javascript_data> with an AJAX GET as follows:
$.get( "/getmethod/<javascript_data>" );
The second one by using an AJAX POST request:
$.post( "/postmethod", {
javascript_data: data
});
Where javascript_data is either a JSON dict or a simple value.
In case you choose JSON, make sure you convert it to a dict in Python:
json.loads(jsdata)[0]
Eg.
GET:
#app.route('/getmethod/<jsdata>')
def get_javascript_data(jsdata):
return json.loads(jsdata)[0]
POST:
#app.route('/postmethod', methods = ['POST'])
def get_post_javascript_data():
jsdata = request.form['javascript_data']
return json.loads(jsdata)[0]
If you need to do it the other way around, pushing Python data down to Javascript, create a simple GET route without parameters that returns a JSON encoded dict:
#app.route('/getpythondata')
def get_python_data():
return json.dumps(pythondata)
Retrieve it from JQuery and decode it:
$.get("/getpythondata", function(data) {
console.log($.parseJSON(data))
})
The [0] in json.loads(jsdata)[0] is there because when you decode a JSON encoded dict in Python, you get a list with the single dict inside, stored at index 0, so your JSON decoded data looks like this:
[{'foo':'bar','baz':'jazz'}] #[0: {'foo':'bar','baz':'jazz'}]
Since what we need is the just the dict inside and not the list, we get the item stored at index 0 which is the dict.
Also, import json.
.html
... id="clickMe" onclick="doFunction();">
.js
function doFunction()
{
const name = document.getElementById("name_").innerHTML
$.ajax({
url: '{{ url_for('view.path') }}',
type: 'POST',
data: {
name: name
},
success: function (response) {
},
error: function (response) {
}
});
};
.py
#app.route("path", methods=['GET', 'POST'])
def view():
name = request.form.get('name')
...
im new in coding, but you can try this:
index.html
<script>
var w = window.innerWidth;
var h = window.innerHeight;
document.getElementById("width").value = w;
document.getElementById("height").value = h;
</script>
<html>
<head>
<!---Your Head--->
</head>
<body>
<form method = "POST" action = "/data">
<input type = "text" id = "InputType" name = "Text">
<input type = "hidden" id = "width" name = "Width">
<input type = "hidden" id = "height" name = "Height">
<input type = "button" onclick = "myFunction()">
</form>
</body>
</html>
.py
from flask import Flask, request
app = Flask(__name__)
html = open("index.html").read()
#app.route("/")
def hello():
return html
#app.route("/data", methods=["POST", "GET"])
def data():
if request.method == "GET":
return "The URL /data is accessed directly. Try going to '/form' to submit form"
if request.method == "POST":
text = request.form["Text"]
w = request.form["Width"]
h = request.form["Height"]
//process your code
return //value of your code

Categories

Resources