I am very new to regex, and I want to extract some data from API responses.
I have a response from my API call:
{
server: 'nginx/1.21.0',
date: 'Mon, 25 Apr 2022 11:34:20 GMT',
'content-type': 'text/html; charset=UTF-8',
'transfer-encoding': 'chunked',
'x-powered-by': 'PHP/7.4.16',
expires: 'Sun, 19 Nov 1978 05:00:00 GMT',
'cache-control': 'no-store, no-cache, must-revalidate, max-stale=0, private, post-check=0, pre-check=0',
'x-content-type-options': 'nosniff',
'content-security-policy': "frame-ancestors 'self';",
'set-cookie': [
'SSESS8b04132f4690dc8ceb7837d5134a5519=ZbgVsmpp6DBRWwNCRMm5DTpfEoKffsNM7l8ILoCWkug; path=/; domain=test.net; secure; HttpOnly',
'su=1; expires=Thu, 22-Apr-2032 11:34:20 GMT; Max-Age=315360000; path=/; domain=test.net',
'lt=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; test.net',
'dtCookie=v_4_srv_10_sn_6364C36040ADFF47B91A838BB8B2CADD_perc_100000_ol_0_mul_1_app-3Aea7c4b59f27d43eb_1; Path=/; Domain=test.net',
'870be6fb89fc78ad2ca0eca3828e3019=7055c78f3c2b7311a5c77934caad6939; path=/; HttpOnly; Secure; SameSite=None'
],
'x-oneagent-js-injection': 'true',
'x-ruxit-js-agent': 'true',
'server-timing': 'dtSInfo;desc="0", dtRpid;desc="493206686"',
connection: 'close'
}
I want to extract SSESS8b04132f4690dc8ceb7837d5134a5519 & ZbgVsmpp6DBRWwNCRMm5DTpfEoKffsNM7l8ILoCWkug which are variables, and every single run that are different values.
I manage to construct a group that is capturing the whole key/value, but still when I try to console it, doesn't work.
https://regex101.com/r/2dKW4s/1
try1 as:
var matches1 = loginModern.headers.match(new RegExp("(SSESS.*)=(.+?);\s*path"));
console.log(matches1[2]);
result output: TypeError: loginModern.headers.match is not a function
try2:
const regex1 = new RegExp('(SSESS.*)=(.+?);\s*path');
var matches1 = regex1.exec(loginModern.headers)
console.log(matches1);
null
How to fetch SSESS8b04132f4690dc8ceb7837d5134a5519 & ZbgVsmpp6DBRWwNCRMm5DTpfEoKffsNM7l8ILoCWkug
from the above response, having in mind they are changing variables?
Any help is appreciated
Sandbox: https://codesandbox.io/s/jwdgmp
const setCookieArray = loginModern.headers["set-cookie"];
const stringContainingSSESS = setCookieArray.find((item) =>
item.includes("SSESS")
);
const matches = stringContainingSSESS.match(/(SSESS[a-zA-Z0-9]*)=(.+?);/);
console.log("key", matches[1]);
console.log("value", matches[2]);
I ran the regexp you provided against the loginModern.headers and it had some issues matching up. This regexp is more accurate
Related
I have the following subdomains:
stream.example.com
sub.example.com
Both domains have SSL certificates and are valid.
I am using videoJS 7.6.6 which has http_streaming library.
On sub.example.com , there is a video tag which sets a dash manifest as source containing links to stream.example.com.VideoJS needs to include laravel cookies from sub.example.com when making a request to stream.example.com links but this is not happening and when i download the HAR result from developer console i see empty cookies in the request.
My VideoJS HTML
<video-js id="player" class="video-js vjs-big-play-centered">
<source src="data:application/dash+xml;charset=utf-8;base64,......." type="application/dash+xml" crossorigin="use-credentials">
</video-js>
The mainifest is valid and it contains stream.example.com urls
VideoJS
player = window.player = videojs('player', {
html5: {
hls: {
withCredentials: true
}
},
controls : true,
fluid: true,
controlBar: {
children: ['playToggle', 'volumePanel', 'currentTimeDisplay', 'timeDivider', 'durationDisplay', 'progressControl', 'liveDisplay', 'seekToLive', 'remainingTimeDisplay', 'customControlSpacer', 'playbackRateMenuButton', 'chaptersButton', 'descriptionsButton', 'subsCapsButton', 'audioTrackButton', 'settingMenuButton', 'qualitySelector','fullscreenToggle']
},
preload : 'auto',
poster : '',
});
player.hotkeys({
volumeStep: 0.1,
seekStep: 5,
alwaysCaptureHotkeys: true
});
var myplugin = window.myplugin = player.myplugin();
}(window, window.videojs));
stream.example.com has the following headers when i view a video link in a browser tab.
accept-ranges: bytes
access-control-allow-credentials: 1
access-control-allow-headers: Origin, X-Requested-With, Content-Type, Content-Length, Authorization,Range
access-control-allow-methods: GET, PUT, POST, DELETE, OPTIONS
access-control-allow-origin: https://sub.example.com
access-control-max-age: 86400
cache-control: private, max-age=18350
content-length: 69688791
content-range: bytes 0-69688790/69688791
content-type: video/mp4
I downloaded the HAR request to see how videoJS is making the request
{
"startedDateTime": "2020-03-15T07:53:57.647Z",
"time": 1.1023430000004737,
"request": {
"method": "GET",
"url": "https://stream.example.com/s/......",
"httpVersion": "",
"headers": [
{
"name": "Referer",
"value": "https://sub.example.com/"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36"
},
{
"name": "DNT",
"value": "1"
},
{
"name": "Range",
"value": "bytes=741-2044"
}
],
"queryString": [
{
"name": "u",
"value": "....."
}
],
"cookies": [], // <-- The cookies are EMPTY
"headersSize": -1,
"bodySize": 0
},
Edit 1
I am already sharing cookies in laravel by adding the following in .env
SESSION_DOMAIN = .example.com
The cookie domain for sub.example.com show .example.com but no cookie for stream.example.com
Edit 2
The response to videojs options request for stream.example.com are shown below
HTTP/2 204 No Content
server: nginx
cache-control: no-cache, private
date: Sat, 21 Mar 2020 06:19:26 GMT
access-control-allow-origin: https://sub.example.com
access-control-allow-methods: GET, POST, HEAD, OPTIONS
access-control-allow-headers: Origin, X-Requested-With, Content-Type, Content-Length, Authorization,Range
access-control-allow-credentials: 1
access-control-max-age: 86400
set-cookie: XSRF-TOKEN=eyJpdiI6ImM4czZNVFRRbWF1emFONXlVMjBGWkE9PSIsInZhbHVlIjoiazVDMUNIR2NqXC9QVUpJdjA3S2lHQ2pKdkJFeHpZdGVodHQ5XC9nZ3JHYVQyUk50V2cxdkQrZ1wvV3ZsOEpDVUhBSiIsIm1hYyI6IjUwYjk4ZjYyZDJmNjg1ZjU4YTg2MDE5ZGNkYmZlOTk5NWVmNTE5ZTRjY2Q1YzQ0ZDI3MzEyNWQ0YmExMzVjZGIifQ%3D%3D; expires=Sat, 21-Mar-2020 10:19:26 GMT; Max-Age=14400; path=/; domain=.example.com
set-cookie: laravel_session=eyJpdiI6IkZvZk9vK2J3YVVhQ2Q4VXpTZjZXN3c9PSIsInZhbHVlIjoiNHZId3orR3dQRDRiOXVFVitKR21NU21DbnVFXC9IcFMxaDFsUXRRUG9VQkFHZnNSdVpRSFBaaHJ5cXdGZDJObUgiLCJtYWMiOiI5ZjllY2IwZjFiNzkxYWMxNTI2ZTFiZWU5OTA4YjNjNzIxZWNkMTBiZjY0ZWQzNDBkMzg5MTEzYjM2MjQ4ODk1In0%3D; path=/; domain=.example.com
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
X-Firefox-Spdy: h2
I assume you need to make the cookies valid for all subdomains, "share" them, as described how to share cookie between subdomain and domain:
if you use the following, it will be usable on both domains:
Set-Cookie: name=value; domain=example.com
Try an OPTIONS request to "url": "stream.example.com/s......" and see if you get the correct response cors headers as above? You can edit and resend the request through Firefox Developer tools.
All other look good.
Set cookies for cross origin requests
Apparently you can do this now:
const player = videojs(element, options)
player.crossOrigin('anonymous')
I'm currently using Node.js to serve a webpage that takes in user inputs which are stored on a mongodb server. The web page also displays specified or all user inputs entered. I'm trying to figure how to pass the user inputs from node.js to the <p> element.
In my node.js file I am responding with the user data as a string like so:
response.writeHead(200, {"Content-Type": "text/plain"});
response.write(stringifyMongoDBCollection(user_data_collection));
response.end();
When I do this, this re-directs the client to display the content as text/plain which I expected. The next step is to update just the content of <p>. How can I do this? I thought about re-serving the entire html content with the new populated <p> but that would make all current user inputs disappear...
The user data would be a mongodb collection array and look like this:
[ { _id: 5dda17065f7e9b64282e7291,
date: 'Sat Nov 23 2019 21:37:10 GMT-0800 (Pacific Standard Time)',
field: '127' },
{ _id: 5dda18ecf330d521a035c444,
date: 'Sat Nov 23 2019 21:45:16 GMT-0800 (Pacific Standard Time)',
field: 125},
{ _id: 5dda1951f330d521a035c445,
date: 'Sat Nov 23 2019 21:46:57 GMT-0800 (Pacific Standard Time)',
field: '111' } ]
You could do something like this.
In Node section
res.status(200).send(stringifyMongoDBCollection(user_data_collection));
Client side
function getContent() {
$.ajax({
url: "https://jsonplaceholder.typicode.com/todos",
success: function (res) {
if (res) {
res = res.slice(0, 5); // limiting data to 5
var val = '';
res.forEach(todo => {
val += '<p><b>Title:</b>' + todo.title + ' <b>Completed:</b> ' + todo.completed + '</p>';
});
}
$("#content").html(val);
},
error: function () {
var val = '<p>Error in loading content</p>'
$("#content").html(val);
}
});
}
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<button onclick="getContent()">Get Content</button>
<h1>Todo</h1>
<div id="content"></div>
References
jQuery ajax() Method
Node.js - Response Object
AJAX Introduction
This question already has answers here:
How can I access and process nested objects, arrays, or JSON?
(31 answers)
Closed 6 years ago.
I am trying to consume the data source from a proxy service, and the data source has the following structure:
{
"Items": {
"ItemSection": [
{
"Food": {
"Name": "Potato Chips",
"itemID": "24501"
},
"category": "Snack",
"description": "Some description",
"available": "Shop A"
},
{
"Food": {
"Name": "Cookie",
"itemID": "24510"
},
"category": "Snack",
"description": "Some description in here.",
"available": [
"Shop A",
"Shop B"
]
},
Below is the response headers :
Status Code: 200 OK
Access-Control-Request-Method: OPTIONS, GET
Cache-Control: public, max-age=321
Content-Length: 42158
Content-Type: application/json
Date: Thu, 29 Sep 2016 20:45:49 GMT
Expires: Thu, 29 Sep 2016 20:51:11 GMT
Last-Modified: Thu, 29 Sep 2016 20:31:11 GMT
Server: Microsoft-IIS/8.5
Vary: *
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
access-control-allow-credentials: true
access-control-allow-headers: Authorization, Content-Type
access-control-allow-origin
and this is how I consume the data source with JavaScript:
function getItems() {
var uri = "proxy url goes here";
var xhr = new XMLHttpRequest();
xhr.open("GET", uri, true);
xhr.onload = function () {
var resp = JSON.parse(xhr.responseText);
showItems(resp.value);
}
xhr.send(null);
}
function showItems(item) {
var tableContent = "<tr class='ItemTitle'><td>Item Lists</td></tr>\n";
for (var i = 0; i < item.length; ++i) {
var record = item[i];
tableContent += "<tr><td>" + record.Name + "</td></tr>\n";
}
document.getElementById("showItem").innerHTML = tableContent;
}
The data source doesn't show up as I run the page, all I got is a blank page with the title Items displayed only. But if I replace the code inside xhr.onload = function() {} to this:
var version_d = document.getElementById("ItemLists");
version_d.innerHTML = xhr.responseText;
Then all data source will be displayed on the page but it just displaying a raw data source and I cannot select what I want to show on the page by using showItems function.
Here's my html page :
<!DOCTYPE html>
<html>
<head>
<title>List</title>
<link rel="stylesheet" type="text/css" href="ShowOrders.css" />
<script src="ShowOrders.js"></script>
<script>
window.onload = getItems;
</script>
</head>
<body>
<h1>Items</h1>
<table id="showItem"></table>
</body>
</html>
Why it is not working with JSON.parse ? am I using the Cross-origin resource sharing (CORS) method wrong ? How can I get it works with out using any external libraries or frameworks (pure JavaScript only) ?
resp.value
JSON.parse directly returns.the parsed object, so resp.value is undefined.
You just want
return resp;
I' m using gapi to send gmail. But if I send the mail to myself, it doesn't appear in my inbox. The most strange, it appears in the 'Sent' folder.
`
function sendMessage(email, callback) {
// Web-safe base64
var base64EncodedEmail = btoa(email).replace(/\//g,'_').replace(/\+/g,'-');
var request = gapi.client.gmail.users.messages.send({
'userId': 'me',
'message': {
'raw': base64EncodedEmail
}
});
request.execute(callback);
}
function send() {
var var to = 'znlswd#gmail.com',
subject = 'Hello World',
content = 'send a Gmail.'
var email = "From: 'me'\r\n"+
"To: "+ to +"\r\n"+
"Subject: "+subject+"\r\n"+
"\r\n"+
content;
sendMessage(email, function () {
console.log(arguments);
});
}
`
The information log in console is:
"[↵ {↵ "id": "gapiRpc",↵ "result": {↵ "id": "1471e09d88000bf7",↵ "threadId": "1471e09d88000bf7",↵ "labelIds": [↵ "SENT"↵ ]↵ }↵ }↵]↵"
And I can see this mail in the 'Sent' folder in Gmail, but I didn't get it in the inbox.
The original of this mail in the 'Sent' folder is as follow:
Received: from 823618323534-124pu7sujfj5olv94rkr0apqdppc6nti.apps.googleusercontent.com
named unknown
by gmailapi.google.com
with HTTPREST;
Wed, 9 Jul 2014 19:11:07 -0700
From: znlswd#gmail.com
To: znlswd#gmail.com
Subject: Hello World
Date: Wed, 9 Jul 2014 19:11:07 -0700
Message-Id: <CADOxe9KLTdYfcb8OxWz+rFSM4tVKuZcVV8HJ4byv95+qPA=+dw#mail.gmail.com>
send a Gmail.
Anybody knows why? Thanks very much!
Edit:
I change the 'To:' in Mime message to another mailbox(not Gmail), and receive the mail correctly. Feeling More confused.
Edit:
I add a 'Cc': field, It is : znlswd#gmail.com(myself), 71848140#qq.com, swdpal2014#gmail.com, ezdiary#163.com . All the other guys(Gmail and other mailboxs) received the mail, but I still couldn't receive the mail from myself.
The original mail I received in ezdiary#163.com is as follow:
Received: from mail-qg0-f44.google.com (unknown [209.85.192.44])
by mx42 (Coremail) with SMTP id XMCowEAJvlfC_L1TjzExBg--.81S3;
Thu, 10 Jul 2014 10:39:03 +0800 (CST)
Received: by mail-qg0-f44.google.com with SMTP id j107so7126918qga.31
for <ezdiary#163.com>; Wed, 09 Jul 2014 19:38:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:from:date:message-id:subject:to:cc:content-type;
bh=zgoFEkSzE1bUNp54umXlfaDEDw5bLf2Ei0uFrgrVZic=;
b=UHOLwuTsFL9yajR0nn/TskbhVKrIfzX4OAsNDno4S2QIvwx83H5dOe2WMyFbL6Plmk
n4z2qzGjturoi1411+PGpgj8rt4Y57aDQpu7tEOMolMmFLJomtCSmagHIHhc0qwU+CRL
rTjO2ztwZoj/ejnqwcmANzgzMMnSxxkcIf3OvXhLm+j+5yHQvPhmGWIapFWJaTN+9gb9
Q47Qkqe5dBrsxOnGhgQnr1orbE5NcLIYumZTH1YfCMZvqIjUtmviUQUKpfhNQD5UtCX0
0J2moKK98Q5Vek4Wti/WtnEqOgNTzHkIL1M90eeAJKelyPu4TQ7G9GJxr1FX+s4WhgrM
xTFA==
MIME-Version: 1.0
X-Received: by 10.224.137.9 with SMTP id u9mr78577439qat.24.1404959937414;
Wed, 09 Jul 2014 19:38:57 -0700 (PDT)
Received: from 823618323534-124pu7sujfj5olv94rkr0apqdppc6nti.apps.googleusercontent.com
named unknown by gmailapi.google.com with HTTPREST; Wed, 9 Jul 2014 19:38:56 -0700
From: znlswd#gmail.com
Date: Wed, 9 Jul 2014 19:38:56 -0700
Message-ID: <CADOxe9KF+VjMFw1bSkFd09RVsa7c7-pz9OguCFovOUemW4ZZbg#mail.gmail.com>
Subject: Hello World
To: znlswd#gmail.com
Cc: 71848140#qq.com, swdpal2014#gmail.com, ezdiary#163.com
Content-Type: text/plain; charset=UTF-8
X-CM-TRANSID:XMCowEAJvlfC_L1TjzExBg--.81S3
Authentication-Results: mx42; spf=pass smtp.mail=znlswd#gmail.com; dki
m=pass header.i=#gmail.com
X-Coremail-Antispam: 1Uf129KBjDUn29KB7ZKAUJUUUUU529EdanIXcx71UUUUU7v73
VFW2AGmfu7bjvjm3AaLaJ3UbIYCTnIWIevJa73UjIFyTuYvjxUSeT5DUUUU
send a Gmail.
Edit: the original mail in swdpal2014#gmail.com is as follow:
Delivered-To: swdpal2014#gmail.com
Received: by 10.194.121.101 with SMTP id lj5csp1399131wjb;
Wed, 9 Jul 2014 19:38:58 -0700 (PDT)
Return-Path: <znlswd#gmail.com>
Received-SPF: pass (google.com: domain of znlswd#gmail.com designates 10.224.137.9 as permitted sender) client-ip=10.224.137.9
Authentication-Results: mr.google.com;
spf=pass (google.com: domain of znlswd#gmail.com designates 10.224.137.9 as permitted sender) smtp.mail=znlswd#gmail.com;
dkim=pass header.i=#gmail.com
X-Received: from mr.google.com ([10.224.137.9])
by 10.224.137.9 with SMTP id u9mr68151154qat.24.1404959937949 (num_hops = 1);
Wed, 09 Jul 2014 19:38:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:from:date:message-id:subject:to:cc:content-type;
bh=zgoFEkSzE1bUNp54umXlfaDEDw5bLf2Ei0uFrgrVZic=;
b=UHOLwuTsFL9yajR0nn/TskbhVKrIfzX4OAsNDno4S2QIvwx83H5dOe2WMyFbL6Plmk
n4z2qzGjturoi1411+PGpgj8rt4Y57aDQpu7tEOMolMmFLJomtCSmagHIHhc0qwU+CRL
rTjO2ztwZoj/ejnqwcmANzgzMMnSxxkcIf3OvXhLm+j+5yHQvPhmGWIapFWJaTN+9gb9
Q47Qkqe5dBrsxOnGhgQnr1orbE5NcLIYumZTH1YfCMZvqIjUtmviUQUKpfhNQD5UtCX0
0J2moKK98Q5Vek4Wti/WtnEqOgNTzHkIL1M90eeAJKelyPu4TQ7G9GJxr1FX+s4WhgrM
xTFA==
MIME-Version: 1.0
X-Received: by 10.224.137.9 with SMTP id u9mr78577439qat.24.1404959937414;
Wed, 09 Jul 2014 19:38:57 -0700 (PDT)
Received: from 823618323534-124pu7sujfj5olv94rkr0apqdppc6nti.apps.googleusercontent.com
named unknown by gmailapi.google.com with HTTPREST; Wed, 9 Jul 2014 19:38:56 -0700
From: znlswd#gmail.com
Date: Wed, 9 Jul 2014 19:38:56 -0700
Message-ID: <CADOxe9KF+VjMFw1bSkFd09RVsa7c7-pz9OguCFovOUemW4ZZbg#mail.gmail.com>
Subject: Hello World
To: znlswd#gmail.com
Cc: 71848140#qq.com, swdpal2014#gmail.com, ezdiary#163.com
Content-Type: text/plain; charset=UTF-8
send a Gmail.
So to be clear, sending mail works as it sends correctly to others. In the case of sending mail to yourself (why exactly are you doing this? is this really needed as opposed to say just using messages.insert?) then it appears in only SENT but not also INBOX label. You only get one copy of the mail, that's the same irregardless of API or using web UI. The problem is that the web UI also puts that message in INBOX as well as SENT. (As you can tell, sending to yourself is a bit of a special case due to things like dupe-detection based on the Message-Id header, etc.)
Answer the question myself. First I tried using messages.insert. However, if I set the 'From' the same as the 'To'(my email address), the mail will still only be in the 'SENT' folder. the right way is using the message.modify api to add 'INBOX' and 'UNREAD' label to the mail sent.
gapi.client.request({
path: "gmail/v1/users/me/messages/" + messageid + "/modify",
method: "POST",
body: "{\"addLabelIds\": [\"UNREAD\",\"INBOX\"]}",
callback: function() {
console.log("gmail sent to self");
return console.log(arguments);
}
})
The 'messageid' refer to the id return by the send api.
An untested idea: you can try adding an X-Gmail-Labels: Inbox header to the messages.send REST call in hopes that Gmail will add the label. I saw this header when doing a Google Takeout of some labels in GMail to mbox files.
While requesting HTTP responses with Node.js and importing these into MongoDB, I noticed one or two URLs will have headers that contain illegal characters (since they are being used keys) which will crash the entire script as I try to import into MongoDB. An example is below:
{
"url": "divensurf.com",
"statusCode": 200,
"headers": {
"x-varnish": "2236710953 2236710300",
"vary": "Accept-Encoding,Cookie,X-UA-Device",
"cache-control": "max-age=7200, must-revalidate",
"x-cache": "V1HIT 2",
"content-type": "text/html; charset=UTF-8",
"page.ly": "v4.0",
"x-pingback": "http://divensurf.com/xmlrpc.php",
"date": "Thu, 21 Mar 2013 19:40:59 GMT",
"transfer-encoding": "chunked",
"via": "1.1 varnish",
"connection": "keep-alive",
"last-modified": "Thu, 21 Mar 2013 19:40:57 GMT",
"age": "2"
}
}
The header/key "page.ly" would crash the script, since it contains an illegal character .. Are there any ways to sanitize this key/header which is enclosed in a quote by removing these illegal characters before I import this document into MongoDB?
Below is the code in which I request responses:
(function (i){
http.get(options, function(res) {
var obj = {};
obj.url = hostNames[i];
obj.statusCode = res.statusCode;
obj.headers = res.headers;
db.scrape.save(obj); // imports headers into MongoDB
}).on('error',function(e){
console.log("Error: " + hostNames[i] + "\n" + e.stack); // prints error stack onto console
})
})(i);
For example, it would be from "page.ly" to "pagely"
EDIT: SOLVED. Check Gael's answer.
obj.headers={};
for(var item in res.headers){
obj.headers[ item.replace(/\./,'')] = res.headers[item];
}