Develop Reference

Python submit tags with requests

So, i need to set tags for the video that i upload, but i can't get it to work.
This is the tags form:
<input id="video_keywords" class="ui-autocomplete-input" type="text" onchange="checkTags()" onkeyup="checkTags()" size="72" name="keywords" autocomplete="off" role="textbox" aria-autocomplete="list" aria-haspopup="true">
And this is my code so far:
payload = {
'keywords': 'tag1 tag2 tag3',
'APC_UPLOAD_PROGRESS': apc,
'message' : '',
}
files = {'upload_file' : (open("D:\\CPA\\videos\\watermarked\\test.avi", 'rb'))}
url = 'http://upload.site.com/account/uploads/submit?video_type=other'
r = s.post(url,data = payload,files = files,headers = headers)
I've tried lots of different things, formatting the tags as a list with and without a space between, i tried encoding the string into json, the website still returns a message that the tags need to be submitted.
Anyone has any idea as to what i'm doing wrong?
And another thing, while inspecting the http headers/requests in a browser, i've noticed that whenever i type tags i receive a GET request from the site, looking like this :
GET /account/title_blacklist/?title=tag1%20tag2 (maybe this is useful?)

There are all sorts of potential reasons this isn't working, but there's not enough information here to be certain why. Here are a couple potential reasons:
Most likely, the checkTags() Javascript function is performing some sort of validation on the tags. Judging by the GET request you showed, it's doing some AJAX to the server, and it's probably looking for blacklisted tags. Or, it could just be sending them to the server to have their format "validated" (less likely). In order to test this out, I'd try playing around with some GET requests to that URL, putting different tags in the GET parameters. By seeing what comes out, you might get a better idea of what tags you're allowed to send. You might also try using your browser's developer tools (like Chrome's) to view the POST data as you send it to this form.
It's also possible that there is some sort of CSRF token in the form. The server generates such a token when it sends the form, and it's valid for a short period of time afterwards. It is sent back to the server with the form data. The server will check the token for validity when it receives the form. It exists to prevent other people (especially malicious websites) from submitting form data to their form. It could be that, for security reasons, the server won't mention the CSRF token in its response, and instead says that the tags are the reason for the error.
All of these possibilities can be checked by examining the requests the webpage sends in the developer tools, and also examining the entire source of the <form>, not just that specific <input>. Hopefully this helps!

I solved the issue.
The form was actually supposed to be called with 'tags', and not by it's name.
It's kinda weird, but it's solved now.

IE8 - IE10 cross domain JSONP cookie headache

Due to decisions that are completely outside of my control, I am in the following situation:
I have a product listing on catalog.org
Clicking the "Add to Cart" button on a product makes an AJAX JSONP request to secure.com/product/add/[productKey], which saves the cart record to the database, sets a cookie with the cart ID, and returns a true response (or false if it failed)
Back on catalog.org, if the response is true, another AJAX JSONP request is made to secure.com/cart/info, which reads the cart ID cookie, fetches the record, and returns the number of items in the cart
Back on catalog.org once again, the response is read and an element on the page is updated showing the number of items in the cart (if any)
At this point, clicking the "Go to Cart" button on catalog.org displays the cart summary on secure.com
This works beautifully in Firefox 17, Chrome 32 and IE 11. It also works in IE8 - IE10 on our development and test environments, where catalog.org is catalog.development.com and catalog.test.com and secure.com is secure.development.com and secure.test.com respectively.
However, after we deployed to production, this stopped working in IE8 - IE10. After adding a product to the cart, the number of items in the cart is updated successfully on catalog.org. Then, after clicking the "Go to Cart" button on catalog.org, the cart summary on secure.com shows nothing because it can't read the cookie. Going to Cache > "View cookie information" in IE develeoper tools shows no cart ID cookie. It should be there, just like it is there in other browsers and in our development and test environments.
I believe what's happening is IE is blocking third party cookies. We have added a P3P compact policy header to all requests on secure.com, but the cookie is still not being set. The header we are setting is:
P3P: CP="CAO PSA OUR"
Why doesn't adding the compact policy header fix this in IE8 - IE10? How can I fix this to work in all versions of IE?
Solution
There are several good ideas posted below. I accepted #sdecima's because it sounded the most promising. We ended up combining some of these ideas but managed to avoid XDomainRequest:
Clicking the "Add to Cart" button on a product makes an AJAX JSONP
request to secure.com/product/add/[productKey], which saves the cart
record to the database, sets a cookie with the cart ID, and returns a
true response (or false if it failed)
We changed the action at secure.com/product/add to return a JSON object with a boolean indicating success or failure and the cart ID.
Back on catalog.org, if the response is true, another AJAX JSONP
request is made to secure.com/cart/info, which reads the cart ID
cookie, fetches the record, and returns the number of items in the
cart
We changed the callback function to check for both properties in the response object. If success is true and the cart ID is present, we create a hidden iframe on the page. The src attribute of the iframe is set to a new endpoint we added to secure.com. This action accepts a cart ID parameter and saves the cart ID cookie. We no longer need to save the cookie in the secure.com/product/add action.
Next, we changed the action at secure.com/cart/info to accept a cart ID parameter. This action will use the cart ID parameter if present to fetch the cart information, otherwise it will still attempt to read the cookie. This extra check would be unnecessary if we could guarantee that the iframe had finished loading and the cookie had been saved on secure.com, but we have no way of knowing when the iframe has finished loading on catalog.org due to browser security restrictions.
Finally, the P3P header CP="CAO PSA OUR" is still required for this to work in IE7 - IE10. (Yes, this works in IE7 now too :)
We now have a solution (albeit an incredibly complex one) for saving and accessing cross domain cookies that works in all major browser, at least as far back as we can reliably test.
We will probably refactor this some more. For one thing, the second AJAX JSONP request to secure.com/cart/info is redundant at this point since we can return all the information we need in the original request to secure.com/product/add action (a side benefit of changing that action to return a JSON object - plus we can return an error message indicating exactly why it failed if there was an error).

In short
Cookies will NOT go through a cross-origin request on IE 8 and 9. It should work on IE 10 and 11 though.
IE 8 and 9
On IE8/9 XMLHttpRequest partially supports CORS, and cross-origin requests are made with the help of the XDomainRequest object which does NOT send cookies with each request.
You can read more about this on the following official MSDN Blog post:
http://blogs.msdn.com/b/ieinternals/archive/2010/05/13/xdomainrequest-restrictions-limitations-and-workarounds.aspx
Particularly this part:
5 . No authentication or cookies will be sent with the request
In order to prevent misuse of the user’s ambient authority (e.g.
cookies, HTTP credentials, client certificates, etc), the request will
be stripped of cookies and credentials and will ignore any
authentication challenges or Set-Cookie directives in the HTTP
response. XDomainRequests will not be sent on previously-authenticated
connections, because some Windows authentication protocols (e.g.
NTLM/Kerberos) are per-connection-based rather than per-request-based.
IE 10+
Starting with IE10, full CORS support was added to XMLHTTPRequest and it should work fine with a correct Access-Control-Allow-Origin header property on the response from the server (that wishes to set the cookie on the browser).
More about this here:
http://blogs.msdn.com/b/ie/archive/2012/02/09/cors-for-xhr-in-ie10.aspx
And here:
http://www.html5rocks.com/en/tutorials/cors/
Workarounds on IE 8 and 9
The only way to go around this on IE8/9 is, quoting the same MSDN post as above:
Sites that wish to perform authentication of the user for cross-origin
requests can use explicit methods (e.g. tokens in the POST body or
URL) to pass this authentication information without risking the
user’s ambient authority.

Bottom line: third party cookies are commonly blocked by privacy/advertisement blocking extensions and should be considered unreliable. You'll be shooting yourself in the foot leaving it in production.
The syntax suggests that the endpoint has ambitions to one day become RESTful. The only problem with that is using cookies, which throws the whole "stateless" concept out of the window! Ideally, changes should be made to the API. If you are not integrating with a third party (i.e. "secure.com" is operated by your company) this is absolutely the correct way to deal with the issue.
Move the cartId out of the secure.com cookie into its querystring:
secure.com/product/add/9876?cartId=1234 //should be a POST
Where to get a valid cartId value? We can persist it in some secure-com-cart-id cookie set for catalog domain, which will avoid any cross-domain issues. Check that value and, if present, append to every secure.com request as above:
$.post('secure.com/product/add/9876', { //needs jQuery.cookie
cartId: $.cookie('secure-com-cart-id')
});
If you don't have a valid cartId, treat it as a new user and make the request without the parameter. Your API should then assign a new id and return it in the response. The "local" secure-com-cart-id cookie can then be updated. Rinse and repeat.
Voila, you've just persisted an active user cart without polluting API calls with cookies. Go yell at your architect. If you can't do that (changing API syntax or yelling), you'll have to set up a tunnel to secure.com endpoint so that there will be no cross-domain request - basically something sitting at catalog.org/secure-com-endpoint which will channel the requests to secure.com verbatim. It's a workaround specifically to avoid making changes to the API, just don't do it with code and have proper Apache/IIS/F5 rules set up to handle it instead. A quick search comes up with several explanations, this one looks pretty good to me.
P.S.: this is a classic XY problem in my opinion. The solution isn't necessarily about persisting 3rd party cookies but about passing necessary parameters to a 3rd party while persisting the data somewhere.

Although a correct solution would be a change of architecture, if you're looking for a quick, temporary solution:
JSONP files are actually just javascript. You could add a line of code to set cookies to the front of your JSONP.
eg instead of:
callback({"exampleKey": "exampleValue"});
Your JSONP could look like:
document.cookie="cartID=1234";
callback({"exampleKey": "exampleValue"});

If you control the DNS records, create a new entry so both servers are in the same domain.

Is there 1 database serving catalog.org and secure.com or can they communicate?
If so then, you got it.
When catalog.org servers a cookie, save it in the db.
When secure.com servers a cookie, save it in the db.
Then you can determine who's cart belongs to which user.
This is a fun problem to consider......Update 2:
When a user goes to catalog.org:
check if he has a cat_org cookie, if not, then:
in catalog.org:
create a key value pair and save in the db {cat_cookie_id, unique_number}
set cat_cookie_id in browser
instruct the browser to ajax visit secure.com/register/unique_number
in secure.com
read unique_number from url
create a secure_cookie id
save in the db {cat_cookie_id, unique_number, secure_cookie_id}
delete unique_number, as this is a one-time use key
Now the db can map cat_cookie_id to secure_cookie_id and vice versa.

request parameters ordering undefined [in multipart/form-data or in general] - what to do?

I am writing a web application that submits a form (one of its fields is mulitpart/form-data, so obviously POST must be used and not GET, since the files might be really big). One of the fields is kinda transaction/upload_id and the other is obviously the file contents. While uploading, a progress bar must be displayed.
The known fact says that the order of parameters is undefined in general, meaning that any of (file content / upload_id) might come first.
Is there any acceptable / recommended way to cause the browser to send the upload_id before sending the file content?
Is it a considered a correct implementation - to expect the upload_id to come first OR there is a better / most common / more correct way to handle the problem? In that case - it would be fantastic to hear some details.
Update: my server-side language is Java/Servlets 3.0

Well, the better answer (without utilizing filters) would be to publish the upload_id(s) as a part of the URL (after '?'), even when issuing a POST request. In that case, they will be always processed ahead of files' contents.

Using servlets as well, and in my case I wanted to run my CSRF filter in my servlet before I started streaming the file: if the filter failed, I can kill the request before I've uploaded my 20gb video file, as opposed to the default PHP implementation where the server only hits your script AFTER its parsed the entire request.
Its been a bit of a hack on my part, but in the couple of cases I've had to do this I've cheated and put the non-file request parameters into the URL and in every case (using pretty much every browser I've tested with) an Iterator over the request parameters on the server (I'm using commons fileupload in streaming mode) has received the non-file request parameters first before the file data was received. Somewhat fragile, but not unworkable.
I'm assuming that if you order your request parameters with the file <input> as the last item you'll get the same behavior.

You shouldn't have to worry about the order in which the parameters are sent. If so, then your server-side code is very brittle.
A multi-part request will contain the field name of every form field that is passed in. Use the name to reference that field regardless of the order it was sent in.
If you are parsing the post body by hand, I suggest you look at existing projects like Apache FileUpload which abstract that away.

Can I suppress the browser’s login prompt on 401 response when using XmlHttpRequest with Twitter

I'm using jQuery's ajax methods to interact with the Twitter REST API.
Their API is a bit annoying, in that some actions will return a 401 HTTP status code. In that case, I just want it to fail, instead of displaying a login box.
I've tried just providing an incorrect user:pass pair in the URL, in case that might suppress the dialog and just get me an error response, but that failed to help.
This question asked exactly the same question, but the answer instead worked out how to not trigger the 401 in the first place.
(In particular this is annoying because it's a bug on their part that's requesting authentication -- the users/show method isn't supposed to be asking for it in the first place.)

Look into suppress_response_codes
It is explained about 1/4 way down the page here:
https://dev.twitter.com/docs/things-every-developer-should-know
From the Twitter documentation (incase it is moved again):
suppress_response_codes: If this
parameter is present, all responses
will be returned with a 200 OK status
code - even errors. This parameter
exists to accommodate Flash and
JavaScript applications running in
browsers that intercept all non-200
responses. If used, it's then the job
of the client to determine error
states by parsing the response body.
Use with caution, as those error
messages may change.

header(Location:) via AJAX not redirecting

First off, let me say that I know this does not seem like an uncommon issue. I do not intend to clutter these forums with questions that have already been answered, but, I have worked through probably about 3 dozen threads with similar issues and have been unable to reach a solution.
The short description of my issue is this: I am trying to redirect after submitting a form using php's header(location:) function, and it's not redirecting. It is a little bit complicated because the form is submitted via AJAX. The steps my code goes through are as follows:
User submits form > form data sent via AJAX > form processing page validates data > if valid, form processing page inserts data into database > if submission is successful, form processing page adds redirect URL to $_SESSION > form processing page returns a 'redirect' variable > javascript/AJAX checks for redirect variable, and refreshes page if found > page header checks $_SESSION for redirect URL, and if found, sets appropriate headers and exits.
I guess the first thing I want to ask is, is this the right way of going about this? It seems to me that there should be a simpler way of doing this. It's obviously much simpler to pass the redirect URL to the javascript and do a window.location redirect, but I read that it's much better/more secure to handle that on the server side, so I'm trying to do that.
Assuming I'm going about this in the right direction, what could be going wrong here? I've checked for leading and trailing whitespace, BOM characters, I've even tried output buffering and I still have the same issue.
What happens on form submission is, the page refreshes, but it returns to the original form page rather than the redirect url. I have turned on the most detailed error reporting and get no errors at all.
One thing that may or may not be of interest, I have an error_log function set up to log all headers to a file right after I set the Location: header. When I redirect outside of AJAX (which works), the accept: header is set to html, but when I try to do it via AJAX, the accept header is set to JSON. Could that cause a problem?
Thank you so much for taking the time, and again, apologies if this is a dumb question. I have used this forum for years and have never once had to post a question on it because it has always solved my problems until now. Thanks in advance!

PHP is too generous to include in your code not only HTML code, but also JavaScript code. I'll explain one thing. When you send data by ajax, it is often difficult return Boolean data (whether true or false, depending on the conditions we work side server with php in some file established in our direction ajax) to give a final answer.
On the other hand, returning to the generosity of PHP, always when we try to validate data server-side and through ajax, we can manipulate the section where work freely to show some response message, a redirect to somewhere on your page or create a session. Anyway, whatever.
What I mean is that in your controller section where you will work with AJAX, you can set different conditions. That is, if the user data are correct, then you could send to call a script from Javascript to redirect him with a location.reload (); and also assign a session automatically. If the user does not have the correct data, then what we should do is return a warning message or alert to the exit AJAX, usually it goes after a:
function (response) {
$ ('.answer').html(response);
}
Note that it is not necessary require, for example, a $ ('.answer').html(response); to return an answer, because ajax only sends data to the server and if there is a reply message well, and if not well. Many times what I do, is to send messages via console, although it is often convenient to do so, because as often work with several conditions and that depends on the result of our request, we will be forced to show some response in some element within our code.
What I advise you is that you work javascript in PHP, and hence redirect to other pages, assign sessions or simply return a message. Remember that an ajax request is not only asynchronous, but repetitive and can send it to call as often as you need. The fact that you sent your ajax call php file and you have returned an answer, does not mean you can go back to work with it once you finish your ajax request.
It is the right way to do what you want, it is often easier to manipulate our server-side code that client side. Greetings.