Develop Reference

Post request header: need fingerprint and re_token to pass it

So im trying to login to anghami.com using post request. I was able to login if I pass all of the values needed.
My issue is im not sure were to find the 2 missing values "re_token" ( I guess its recapthca from google) and "fingerprint"
check my code for working test:
headers = {
'authority': '',
'accept': 'application/json, text/plain, */*',
'accept-language': 'en-US,en;q=0.9',
'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
'origin': 'https://play.anghami.com',
'referer': 'https://play.anghami.com/login',
'sec-ch-ua': '".Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-site',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}
link = 'https://api.anghami.com/gateway.php'
s = requests.Session()
proxies = {
"http": "http://",
}
data = {
'm': 'an',
'u': 'email',
'p': 'password',
'devicename': 'Chrome 104',
're_token':'03ANYolqvEDNfBhyM-qJ77j92_vkw8yt1VtKKc9e6jZFl9mG4sysFOvVZ0LlQWsMecFRWRCMFGG8KAgdWw1S0kUPB-1yW5kfJ8B2XGLnlaW7XAReGvyYpB2WgZeGXPdxlTi0PINbN2Ga9wI2ecF9jltpf7gcUj9MLucb9KDaUYENySmFq2ts5qh9g_2nr6AXx_igsD53xvWPGrGi_n7evy224P7A0NitmjcXKlAKL_rlkkXqbwOd4qbzF_IkTKX6iSNLfb2FFso8S75OKa0dlbfLO_7eY2zU7VzVKa23XWet3RXDED7q8Rx8RKFaO9n_lvbG-PORGCpmajnbWtWoIhEZpY06mt41vx4AoW0JnCtV9Z3v5AsAoM_SIZNawTLVBKyI3iVk9AbsGskh5DZ0DzIQ2Hp_2325fuyhjp2gjW_yUud7DuGVZ9Zn7WjteVnE0Yv4ZQoWx5Z2Hz-s7Qy7G2Acm6WLbuIvS_5JsJsfLYh_hiB_DY79UyKHNpeQtulqS1wMGwHqDFbmfv',
'ngsw-bypass': 'true',
'type': 'authenticate',
'language': 'en',
'lang': 'en',
'web2': 'true',
'fingerprint': 'eyJmcCI6ImE2MzcxYTRjLTU1ODEtNDE2My1iMWRkLTA3NjBkMmI5OWZlYSIsImgiOiI0OTIxZDM1OCJ9',
'angh_type': 'authenticate',
}
auth = HTTPProxyAuth(proxyacc)
s.proxies = proxies
s.auth = auth
ext_ip = s.post(link, data=data, headers=headers)
print (ext_ip.text)
print (ext_ip.url)
How can I create 'fingerprint':
how to get 're_token':

So,
re_token -> recaptcha token(answer)
fingerprint -> base64 encoded string with fingerprint uid and
unknown 'h' param.
For more accurate information needed debug vendor.67cc4b67b66a6114.js this code.
As for the recaptcha, you need to understand what type of captcha is used on the site and, accordingly, find the key and additional information if needed.
To solve it, you can use any ready-made service like anti-captcha.
By the way, it's a bad idea to specify the authority header, since this header is specified automatically and works only with http2.0, otherwise, it betrays you even more.

UrlFetchApp fetch how to retrieve JSESSIONID object for the cookies?

I'm trying to use UrlfetchApp to send a request to a page. Request is good. But returning
<p>Your browser is currently set to block cookies. Please enable cookies in your browser preferences and try again.</p>
in the html body.
Here is the code:
const res1 = UrlFetchApp.fetch('https://url.com', {
method: 'POST',
headers: {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Language': 'en-US,en;q=0.9',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'Content-Type': 'application/x-www-form-urlencoded',
'Cookie': 'JSESSIONID=Dfaefaefaesfsaefgr',
'Origin': 'https://url.com',
'Referer': 'https://url.com',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'same-origin',
'Sec-Fetch-User': '?1',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"'
},
payload: 'payloaddata'});
I don't know how to turn on the cookie when using UrlFetch function. Does anyone has the same issue?
Thanks for any suggestions.
What I have tried:
Seperate Cookie property from headers into a json object.
cookies = {
'JSESSIONID': '778C494754356A23F080849C10F2A851',
'TS01ee6e39': '018c1954d58470c0adb8d6d0df850dba9363aa54c16fc4653b2178278f62a5f0dde96d5780b3338857c8ea3ff92c6e99b9bd3a5867cef327bd847ac05ab9242a7414fa0832',
'X-HR-ClientSessionId': '10_107.162.4.39_1651626597548',
'locale': 'en',
'TS0189a565': '018c1954d5ae31eb4b4d18a85d43414cdcd9158bcc6fc4653b2178278f62a5f0dde96d578016166d716abd84dc80b66117923674fd68b4dd222e33d15361b818320e9dbd03333b4682ebc552fb370cd462eb3e5d2b5cc5273e789e87f2bd772fa800fa9e77744d459169cf3d8594422a3d7ae7968ba33103e373fbcf2c83f38da92d9643e5f9ef6a925938338bef881e38e827bf0a5635126e7297731cc06d71fd1a883702',}
But this does not work, request immediately return session expired. Please try again.
Edit on 05/07/2022:
I'm trying to get the JSESSIONID from fetch request, I can find this object on the browser, But could not find it on the urlfetch return response.

Why can access a site through browser, wheares write simple code to fetch, but get 403 error

Endpoint: https://quizlet.com/webapi/3.2/images/search?query=hello&perPage=2
You guys can try to access this page as Incognito, from my side It works. So I think I can fetch data from that site.
I try to copy the request and run in Javascirpt, Python. However, It doesn't work. I got 403 error.
I also try to use Burp Suite. I can't access this site through Burp's browser.
Moreover, As I tried using incognito so I don't think it is relevant to cookies.
Code sample (JS):
import fetch from "node-fetch";
const response = await fetch(
"https://quizlet.com/webapi/3.2/images/search?query=hello&perPage=2",
{
headers: {
accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-language": "en",
"cache-control": "no-cache",
pragma: "no-cache",
"sec-ch-ua":
'"Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Linux"',
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1",
},
referrerPolicy: "strict-origin-when-cross-origin",
body: null,
method: "GET",
mode: "cors",
credentials: "include",
}
);
const data = await response.status;
console.log(data);
Code Python
import requests
headers = {
'authority': 'quizlet.com',
'pragma': 'no-cache',
'cache-control': 'no-cache',
'sec-ch-ua': '"Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Linux"',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'sec-fetch-site': 'none',
'sec-fetch-mode': 'navigate',
'sec-fetch-user': '?1',
'sec-fetch-dest': 'document',
'accept-language': 'en',
'cookie': 'qi5=i2x3g7y1z9a6%3At3vMoQQig2yLcpN.HKWn; qtkn=7gT4DE7pN9URJ2AFDYeaVe; fs=qzkse0; app_session_id=9781a407-4f37-4c09-8e97-8156f182bb45; search_session=%7B%22search_session_id%22%3A%22-2379864199063990974614477b859794%22%2C%22query%22%3A%22overrated%22%2C%22version%22%3A%221.1.1%22%2C%22platform%22%3A%22WEB%22%2C%22depth%22%3Anull%2C%22target_object_type%22%3A%22QImage%22%7D; __cf_bm=cB7hRf6JbcOFZ2kvQ3W12V4bxXiIgn_kF3n87RcI0h0-1631877048-0-Ac+Hi0pATLgW5N3JjqYa7uc5W4ZfDLOumvmCQixWJIKdcVj7stciFh8cYFVTOpr+q5pM2Q7LrXC/LsffOB6Mh2E=; __cfruid=81f16a673e6117331dd4270b3f4f29111590d7d8-1631877048',
}
params = (
('query', 'hello'),
('perPage', '2'),
)
response = requests.get(
'https://quizlet.com/webapi/3.2/images/search', headers=headers, params=params)
# NB. Original query string below. It seems impossible to parse and
# reproduce query strings 100% accurately so the one below is given
# in case the reproduced version is not "correct".
# response = requests.get('https://quizlet.com/webapi/3.2/images/search?query=hello&perPage=2', headers=headers)
print(response.status_code)
Please help me out. I don't even know how can be that? (browser works, while code doesn't). Thank anyway.

From the python side. I had a look out of interest, as I'm currently developing a REST API and was curious how they where securing it.
Using Wireshark it appears that the "requests" module in python does not handle http requests in the same manor as Chrome/Firefox, which I suspect they are using as a tell to give a captcha.
Anyway switching requests for the httpx module;
pip install httpx
https://www.python-httpx.org/
And changing the headers to replicate Firefox in full;
import httpx
headers = [
('Accept','text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'),
('Accept-Encoding','gzip, deflate, br'),
('Accept-Language','en-GB,en;q=0.5'),
('Cache-Control','max-age=0'),
('Connection','keep-alive'),
('Host','quizlet.com'),
('Sec-Fetch-Dest','document'),
('Sec-Fetch-Mode','navigate'),
('Sec-Fetch-Site','none'),
('Sec-Fetch-User','?1'),
('TE','trailers'),
('Upgrade-Insecure-Requests','1'),
('User-Agent','Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'),
]
params = (
('query', 'hello'),
('perPage', '2'),
)
response = httpx.get('https://quizlet.com/webapi/3.2/images/search', headers=headers, params=params,)
print(response.content)
Gives the following as appose to the captcha page for me;
{
"responses": [{
"models": {
"image": [{
"id": 18957872,
"personId": 16641862,
"timestamp": 1416579222,
"lastModified": 1416579222,
"code": "Gfg5XS88MRmYq8RS",
"license": 1,
"width": 480,
"height": 360,
"flickrId": null,
"flickrOwner": null,
"_legacyUrl": "http://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA.gif",
"_legacyUrlSquare": "http://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA_s.gif",
"_legacyUrlSmall": "http://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA_m.gif",
"_secureLegacyUrl": "https://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA.gif",
"_secureLegacyUrlLarge": "https://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA_b.gif",
"_secureLegacyUrlSquare": "https://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA_s.gif",
"_secureLegacyUrlSmall": "https://o.quizlet.com/cZDE.6rHW7IrGptXSGm8FA_m.gif"
}, {
"id": 9228314,
"personId": 513525,
"timestamp": 1406222781,
"lastModified": 1406222781,
"code": "bPHbzaV7KsGWfuXJ",
"license": 1,
"width": 298,
"height": 232,
"flickrId": null,
"flickrOwner": null,
"_legacyUrl": "http://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA.jpg",
"_legacyUrlSquare": "http://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA_s.jpg",
"_legacyUrlSmall": "http://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA_m.jpg",
"_secureLegacyUrl": "https://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA.jpg",
"_secureLegacyUrlLarge": "https://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA_b.jpg",
"_secureLegacyUrlSquare": "https://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA_s.jpg",
"_secureLegacyUrlSmall": "https://o.quizlet.com/ptqCa7LsKjiVSBVPI3OfTA_m.jpg"
}]
},
"paging": {
"total": 50,
"page": 1,
"perPage": 2,
"token": "UuKKKAkmxv.r4YtwFDuRevZVGAHr"
}
}]
}

jQuery Ajax Form Data settings convert to pure javascript

Found the answer thanks from Patrick Evans:
window.onload = function()
{
var data = new FormData();
data.append("gcd", "gcd");
data.append("name", "name");
ajax({
type: "POST",
url: url,
data: data,
success: function(resopnse)
{
console.log(resopnse);
},
dataType: "json"
});
}
var http_request = new XMLHttpRequest();
function ajax(options) {
http_request.open(options.type || 'GET', options.url, true);
http_request.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
http_request.send(options.data || null);
http_request.onreadystatechange = function() {
if (http_request.readyState == 4) {
if (http_request.status == 200) {
var type = options.dataType || '';
switch (type.toLowerCase()) {
default:
options.success(http_request.responseText);
break;
case 'json':
options.success(JSON.parse(http_request.responseText));
break;
}
}
}
}
}
Here is my Ajax test javascript using jQuery and Pure Javascript Ajax:
window.onload = function()
{
var url = "GRNM";
var data = {
gcd: "gcd",
name: "name"
};
$.ajax({
type: "POST",
url: url,
data: data,
success: function(resopnse)
{
console.log(resopnse);
},
dataType: "json"
});
ajax({
type: "POST",
url: url,
data: data,
success: function(resopnse)
{
console.log(resopnse);
},
dataType: "json"
});
}
Pure Javascript Ajax:
var http_request = new XMLHttpRequest();
function ajax(options) {
http_request.open(options.type || 'GET', options.url, true);
http_request.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8');
http_request.setRequestHeader('X-Requested-With', 'XMLHttpRequest');
http_request.send(JSON.stringify(options.data) || null);
http_request.onreadystatechange = function() {
if (http_request.readyState == 4) {
if (http_request.status == 200) {
var type = options.dataType || '';
switch (type.toLowerCase()) {
default:
options.success(http_request.responseText);
break;
case 'json':
options.success(JSON.parse(http_request.responseText));
break;
}
}
}
}
}
And this is the result:
jQuery: I could get the value data successfully
General:
Request URL: http://gaspc-011:8888/GRNM
Request Method: POST
Status Code: 200
Remote Address: 192.168.1.120:8888
Referrer Policy: strict-origin-when-cross-origin
Response Headers:
Connection: keep-alive
Content-Length: 43
Content-Type: application/json
Date: Fri, 27 Aug 2021 06:20:37 GMT
Keep-Alive: timeout=60
Request Headers:
Accept: application/json, text/javascript, */*; q=0.01
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 17
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: gaspc-011:8888
Origin: http://gaspc-011:8888
Referer: http://gaspc-011:8888/index01
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36
X-Requested-With: XMLHttpRequest
Form Data:
gcd: gcd
name: name
Pure Javascript: I couldn't get the data
General:
Request URL: http://gaspc-011:8888/GRNM
Request Method: POST
Status Code: 400
Remote Address: 192.168.1.120:8888
Referrer Policy: strict-origin-when-cross-origin
Response Headers:
Connection: close
Content-Language: en-US
Content-Type: text/html;charset=Shift_JIS
Date: Fri, 27 Aug 2021 06:20:37 GMT
Transfer-Encoding: chunked
Request Headers:
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 27
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: gaspc-011:8888
Origin: http://gaspc-011:8888
Referer: http://gaspc-011:8888/index01
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36
X-Requested-With: XMLHttpRequest
Form Data:
{"gcd":"gcd","name":"name"}:
My Spring Boot Controller couldn't find the gcd and name parameter that came from pure javascript because the Form Data format is different. I've also tried to use FormData() but couldn't make it work.
My Form Data becomes like this:
Form Data:
------WebKitFormBoundaryLgD8tjkxnVk4hfiE
Content-Disposition: form-data; name: "gcd"
gcd
------WebKitFormBoundaryLgD8tjkxnVk4hfiE
Content-Disposition: form-data; name="name"
name
------WebKitFormBoundaryLgD8tjkxnVk4hfiE--
I've also tried changing http_request.send(JSON.stringify(options.data) || null); to http_request.send(options.data || null); but didn't worked.
How can I achieve the same result as jQuery? How can I pass my var data object to controller using Ajax POST same as jQuery?

You need to supply the correct content-type with the correct content.
If you want to send JSON text you have to use application/json content-type
http_request.setRequestHeader('Content-Type', 'application/json');
http_request.send(JSON.stringify(options.data));
If you want to use the FormData object you need the multipart/form-data content-type
let fd = new FormData();
for(let key in options.data){
fd.append(key,options.data[key]);
}
//don't need to explicitly set content-type when sending FormData
//it will automatically do that
//http_request.setRequestHeader('Content-Type', 'multipart/form-data');
http_request.send(fd);
If you just want to use your object you will need to convert it to one of the previously mentioned methods or create a param string from it and use application/x-www-form-urlencoded content-type
//builds a param=value&param2=value2 type of string from your options.data object
let paramStrings = [];
for(let key in options.data){
paramStrings.push(`${key}=${options.data[key]}`);
}
let data = paramStrings.join('&');
http_request.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8');
http_request.send(data);

Can't set or override on specific header using Angular.js $http

I'm using Angular.js 1.4.8 and Fiddler 4 for debugging my requests.
The following is the request I made using AngularJS $http.
var postRequest = {
method: 'POST',
url: 'https://speech.platform.bing.com/recognize',
headers: {
'Transfer-Encoding': 'chunked',
Expect: '100-continue',
Expect2: 'abc',
Accept: 'application/json;text/xml',
Host: 'region.platform.bing.com',
'Content-Type': 'audio/wav; samplerate=8000',
Authorization: 'auth-token',
'Accept-Language': undefined,
'Accept-Encoding': undefined,
'User-Agent': undefined,
},
params: {
scenarios: 'smd', // 'smd' in the internal sample code,
locale: langString,
'device.os': 'wp7',
version: '3.0',
format: 'json',
},
data: "test"
};
$http(postRequest).then(function (response) {
console.log(response)
});
However, as described in below, in the actual request, there are some missing headers (e.g., Expect, Transfer-Encoding). In addition, there are still automatically added headers by Angular even after I set it as undefined (as guided in official document: https://code.angularjs.org/1.4.8/docs/api/ng/service/$http).
POST https://speech.platform.bing.com/recognize?device.os=wp7&format=json&locale=en-US&scenarios=smd&version=3.0 HTTP/1.1
Expect2: abc
Accept: application/json;text/xml
Content-Type: audio/wav; samplerate=8000
Authorization: 'auth-token'
**Accept-Language: en-US,en;q=0.8,ko;q=0.6,zh-Hans-CN;q=0.4,zh-Hans;q=0.2
Accept-Encoding: gzip, deflate**
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; MSAppHost/3.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240
Host: speech.platform.bing.com
Content-Length: 271491 Connection: Keep-Alive Cache-Control: no-cache
Cookie: MUIDB=39DE0AD21AD46AF2039D02BB1BB26B61
Is there any ways that
Can I add headers which I could not add using 'headers' objects, and
Can I remove the headers automatically added by Angular?
Or is it the Angular bug?

Yes you can. You can use angular interceptor to handle it globally. So for each request, you can automatically remove the unwanted headers and add necessary headers.
$httpProvider.interceptors.push(['$q', '$location', 'localStorageService', function ($q, $location, localStorageService) {
return {
request: function (config) {
config.headers = config.headers || {};
var authData = localStorageService.get('authData');
if (authData) {
config.headers.Authorization = 'Bearer ' + authData.token;
}
return config;
},
response: function (result) {
return result;
},
responseError: function (rejection) {
console.log('Failed with', rejection.status, 'status');
if (rejection.status == 401) {
localStorageService.remove('authData');
window.location.replace(window.location.origin)
}
if (rejection.status == 307) {
$location.url('/SessionExpired');
}
if (rejection.status == 403) {
$location.url('/Forbidden');
}
if (rejection.status == 500) {
$location.url('/InternalServerError');
}
return $q.reject(rejection);
}
}
}])

Setting HTTP Headers
module.run(function($http) {
$http.defaults.headers.common.Authorization = 'Basic YmVlcDpib29w';
});

Beware, other answers can be misleading ... Yes, using the suggested strategies, you can override some headers but not all! ... The reason of this is that, by specification, some headers are reserved and will be controlled only by the browser ... Examples of these reserved headers are:
Accept-Charset
Accept-Encoding
Access-Control-Request-Headers
Access-Control-Request-Method
...
You can find more information about this in the following urls:
AngularJS: Refused to set unsafe header “Access-Control-Request-Headers”
Forbidden Header name for use with XMLHttpRequest