I want to create an Electron app that will use webview to display 3rd party content.
I would like to be able to intercept all requests and responses from this webview. Sometimes I would like to manipulate this content, other times I would like to log it, and other times I’d like to do nothing.
As one example for the responses, maybe a web server will respond with TypeScript code, maybe I want to take that response, and compile it to standard JavaScript.
I have looked into this page but it looks like it is only possible to cancel requests, and manipulate the headers. The WebRequest API doesn't look to fit the needs of my use case since it only allows very minor manipulations of requests and responses.
I have also considered setting up some time of web server that can act as a proxy, but I have concerns about that. I want to maintain user privacy, and I want to ensure that to the web servers that host the 3rd party content it looks like the request is coming from a browser like environment (ex. Electron webview) instead of a server. I know I can manipulate requests with the headers I send and such, but this whole solution is getting to be a lot more complicated, then I would like, but might be the only option.
Any better ways to achieve this, and have more control over the Electron webview?
I think you should look into the Protocol API. It works as a proxy internally.
Say you want the user, when opening http://www.google.com, to see content like you've been conned!:
const { protocol } = require("electron");
const content = new Buffer("you've been conned!");
protocol.interceptBufferProtocol("http", (request, result) => {
if (request.url === "http://www.google.com")
return result(content);
... // fetch other http protocol content and return to the electron
});
There's lots of work to do, compared to the WebRequest API, but it's much simpler than an independent local proxy.
To get the request body of any http network call made by your electron app:
session.defaultSession.webRequest.onBeforeSendHeaders(filter, (details, callback) => {
if (details.uploadData) {
const buffer = Array.from(details.uploadData)[0].bytes;
console.log('Request body: ', buffer.toString());
}
callback(details);
})
Related
My main function is I am creating a link-shortening app. When someone entered a long URL, it will give a short URL. If the user clicked on the short link it will search for the long URL on the DB and redirect it to the long URL.
Meantime I want to get the click count and clicked user's OS.
I am currently using current code :
app.get('/:shortUrl', async (req, res) => {
const shortUrl = await ShortUrl.findOne({short: req.params.shortUrl})
if (shortUrl == null) return res.sendStatus(404)
res.redirect(shortUrl.full)
})
findOne is finding the Long URL on the database using ShortID. I used mongoDB here
My questions are :
Are there multiple redirect methods in JS?
Is this method work if there is a high load?
Any other methods I can use to achieve the same result?
What other facts that matter on redirect time
What is 'No Redirection Tracking'?
This is a really long question, Thanks to those who invested their time in this.
Your code is ok, the only limitation is where you run it and mongodb.
I have created apps that are analytics tracker, handling billion rows per day.
I suggest you run your node code using AWS Beanstalk APP. It has low latency and scales on your needs.
And you need to put redis between your request and mongodb, you will call mongodb only if your data is not yet in redis. Mongodb has more read limitations than a straight redis instance.
Are there multiple redirect methods in JS?
First off, there are no redirect methods in Javascript. res.redirect() is a feature of the Express http framework that runs in nodejs. This is the only method built into Express, though all a redirect response consists of is a 3xx (often 302) http response status and setting the Location header to the redirect location. You can manually code that just as well as you can use res.redirect() in Express.
You can look at the res.redirect() code in Express here.
The main things it does are set the location header with this:
this.location(address)
And set the http status (which defaults to 302) with this:
this.statusCode = status;
Then, the rest of the code has to do with handling variable arguments, handling an older design for the API and sending a body in either plain text or html (neither of which is required).
Is this method work if there is a high load?
res.redirect() works just fine at a high load. The bottleneck in your code is probably this line of code:
const shortUrl = await ShortUrl.findOne({short: req.params.shortUrl})
And, how high a scale that goes to depends upon a whole bunch of things about your database, configuration, hardware, setup, etc... You should probably just test how many request/sec of this kind your current database can handle.
Any other methods I can use to achieve the same result?
Sure there are. But, you will have to use some data store to look up the shortUrl to find the long url and you will have to create a 302 response somehow. As said earlier, the scale you can achieve will depend entirely upon your database.
What other facts that matter on redirect time
This is pretty much covered above (hint, its all about the database).
What is 'No Redirection Tracking'?
You can read about it here on MDN.
I have a proxy, and I've fetched contents of the webpage I need, like https://google.com. However, I need to be able to then also redirect all the other requests for resources to go through the proxy. So, all images and scripts go back through the proxy. Additionally, all links also go back through the proxy. How can I access all of the requests and do this? Would this be through modifying the HTML of the site? Now, I should be able to serve the contents of any dynamic or static site on a localhost, without having certain elements and scripts not loading.
Your question states that you have a proxy, but your answer implies that you are implementing a proxy.
It sounds like you're asking for a Node.js solution, since question was tagged node.js and notice says
The answer needs to detail how to implement this in NodeJS, and provide working code examples.
In Node.js, you can achieve this using HTTP Party's http-proxy library
Install it with npm install http-proxy.
Then match routes using your web framework, and rewrite the requests as needed.
In Express:
const httpProxy = require('http-proxy');
const proxy = httpProxy.createProxyServer({
autoRewrite: true, //rewrites the location host/port
changeOrigin: true, //change "Origin" header
});
app.use('/api/v1/',(req, res, next) => {
//redirect requests from `https://api.your.domain/api/v1/` to `https://api.your.domain/api/v2/`
//subpaths like /api/v1/me will also be changed (e.g. `/api/v2/me`)
proxy.web(req, res, {target: 'https://api.your.domain/api/v2/'}, next );
});
So I found the solution!
Using a service worker I can redirect requests to go back through the forward proxy. This example proxies requests from an old api version to a new one (this isn't what Im doing though. It would add a certain url, like https://api.allorigins.win/get?url= to the beginning):
self.addEventListener('fetch', e => {
const {url} = e.request;
if(url.includes('https://api.your.domain/api/v1/') {
const newUrl = url.replace('/api/v1/', '/api/v2/');
e.respondWith(fetch(newUrl));
}
});
This article essentially fixes my problem: https://betterprogramming.pub/how-to-run-a-proxy-server-inside-your-browser-8b96ea2ef1ea
I have created a short url, let say https://my.short.link/foo, that is pointing to https://my.other.website/bar.
How can I retrieve this url with a javascript method inside the browser? (I'm using angular)
It will depend on how my.short.link handles processing the URL when you request it.
Type 1: A really basic service might use a redirection response (HTTP 302, etc).
Type 2: Or it might use a successful response serving an HTML page with a <meta http-equiv="refresh" ...> tag in it.
Type 3: Or it might use a successful response serving an HTML page doing the redirection via JavaScript.
Browser
If you mean you want to do this in a browser, you might be able to get the information for service Type 1 above, but not for other services.
As evolutionxbox says, you could try fetch, and surprisingly (to me!) if the service handles requests with a simple redirection response, you do see a new URL in the response, but it wont have the hash fragment:
fetch("https://tsplay.dev/wEv6gN")
.then(r => {
if (!r.ok) {
throw new Error(`HTTP error ${r.status}`);
}
for (const [key, value] of r.headers) {
console.log(`${key}: ${value}`);
}
console.log(r.url);
})
.catch(console.error);
That URL, https://tsplay.dev/wEv6gN, redirects to https://www.typescriptlang.org/play?#code/MYewdgzgLgBAJgQygmBeGB5ARgKwKbBQB0AZgE554BeeAFAN4CwAUDGzBCALZ4BcMARgBMAZgA0LdjAA24AOb96AIgRLeSrEoC+E1u2kBLaPwDaS4EoC6uqSBL8lSm+wRksBqGVcBPfmACu0tLObFAAFgZgchD8AkRCuloAlADcLCxQ3gAOePBICAD6ANZ43mgwJd52MJk51YjIacwsJP5ghAbg8CAA6iBkRbRF-A2FlUkwTHps0niwAG4I0v656KMmRZZNUgD0O0QHLFpAA via a 302 ("Moved Permanently") redirect response. In the code above, we only see https://www.typescriptlang.org/play?. But maybe that's enough for your purposes.
But that won't work for types 2 and 3 above. You'd have to be able to read and parse the HTML, and the Same Origin Policy will prevent code running on your origin from accessing that HTML. You could try opening it in an iframe (though I suspect you don't really want to visit the page, you just want to get its URL), but the same issue applies: Even when the iframe goes to the target location, you won't be able to access that because of the SOP.
Node.js Or Similar
If you mean in Node.js or some other non-browser environment, then:
For a Type 1 service, you could do an HTTP request and look at the redirection information.
For a Type 2 service, you could do the HTTP request, parse the HTML, and look for the tag.
For a Type 3, you'd need to use a headless browser (Puppeteer, etc.) to capture where it was going.
I have built a portal which provides access to several features, including trouble ticket functionality.
The client has asked me to make trouble ticket functionality available offline. They want to be able to "check out" specific existing tickets while online, which are then accessible (view/edit) while the user's device is out-of-range of any internet connection. Also, they want the ability to create new tickets while offline. Then, when the connection is available, they will check in the changed/newly created tickets.
I have been tinkering with Service Workers and reviewing some good documentation on them, and I feel I have a basic understanding of how to cache the data.
However, since I only want to make the Ticketing portion of the portal available offline, I don't want the service worker caching or returning cached data when any other page of the portal is being accessed. All pages are in the same directory, so the service worker, once loaded, would by default intercept all requests from all pages in the portal.
How can I set up the service worker to only respond with cached data when the Tickets page is open?
Do I have to manually check the window.location value when fetch events occur? For example,
if (window.location == 'https://www.myurl.com/tickets')
{
// try to get the request from network. If successful, cache the result.
// If not successful, try returning the request from the cache.
}
else
{
// only try the network, and don't cache the result.
}
There are many supporting files that need to be loaded for the page (i.e. css files, js files, etc.) so it's not enough to simply check the request.url for the page name. Will 'window.location' be accessible in the service worker event, and is this a reasonable way to accomplish this?
Use service worker scoping
I know that you mentioned that you currently have all pages served from the same directory... but if you have any flexibility over your web app's URL structure at all, then the cleanest approach would be to serve your ticket functionality from URLs that begin with a unique path prefix (like /tickets/) and then host your service worker from /tickets/service-worker.js. The effort to reorganize your URLs may be worthwhile if it means being able to take advantage of the default service worker scoping and just not have to worry about pages outside of /tickets/ being controlled by a service worker.
Infer the referrer
There's information in this answer about determining what the referring window client URL is from within your service worker's fetch handler. You can combine that with an initial check in the fetch handler to see if it's a navigation request and use that to exit early.
const TICKETS = '/tickets';
self.addEventListener('fetch', event => {
const requestUrl = new URL(event.request.url);
if (event.request.mode === 'navigate' && requestUrl.pathname !== TICKETS) {
return;
}
const referrerUrl = ...; // See https://stackoverflow.com/questions/50045641
if (referrerUrl.pathname !== TICKETS) {
return;
}
// At this point, you know that it's either a navigation for /tickets,
// or a request for a subresource from /tickets.
});
I'm building a chat dashboard and widget with which a customer should be able to put the widget into their page. Some similar examples would be Intercom or Drift.
Currently, the "main" application is written in Meteor.js (it's front end is in React). I've written a <Widget /> component and thrown it inside a /widget directory. Inside this directory, I also have an index.jsx file, which simply contains the following:
import React from 'react';
import ......
ReactDOM.render(
<Widget/>,
document.getElementById('widget-target')
);
I then setup a webpack configuration with an entry point at index.jsx and when webpack is run spits out a bundle.js in a public directory.
This can then be included on another page by simply including a script and div:
<script src="http://localhost:3000/bundle.js" type="text/javascript"></script>
<div id="widget-target"></div>
A few questions:
What is wrong with this implementation? Are their any security issues to be aware of? Both the examples linked earlier seem make use of an iframe in one form or another.
What is the best way to communicate with my main meteor application? A REST API? Emit events with Socket.io? The widget is a chat widget, so I need to send messages back and forth.
How can I implement some sort of unique identifier/user auth for the user and the widget? Currently, the widget is precompiled.
1 What is wrong with this implementation? Are their any security issues to be aware of? Both the examples linked earlier seem make use of an iframe in one form or another.
As #JeremyK mentioned, you're safer within an iFrame. That being said, there's a middle route that many third parties (Facebook, GA, ...) are using, including Intercom:
ask users to integrate your bundled code within their webpage. It's then up to you to ensure you're not introducing a security vulnerability on their site. This code will do two things:
take care of setting up an iframe, where the main part of your service is going to happen. You can position it, style it etc. This ensure that all the logic happening in the iframe is safe and you're not exposed.
expose some API between your customer webpage and your iframe, using window messaging.
the main code (the iframe code) is then loaded by this first script asynchronously, and not included in it.
For instance Intercom ask customers to include some script on their page: https://developers.intercom.com/docs/single-page-app#section-step-1-include-intercom-js-library that's pretty small (https://js.intercomcdn.com/shim.d97a38b5.js). This loads extra code that sets the iFrame and expose their API that will make it easy to interact with the iFrame, like closing it, setting user properties etc.
2 What is the best way to communicate with my main meteor application? A REST API? Emit events with Socket.io? The widget is a chat widget, so I need to send messages back and forth.
You've three options:
Build your widget as an entire Meteor app. This will increase the size of the code that needs to be loaded. In exchange for the extra code, you can communicate with your backend through the Meteor API, like Meteor.call, get the reactivity of all data (for instance if you send a response to a user through your main Meteor application, the response would pop up on the client with no work to do as long as they are on the same database (no need to be on the same server)), and the optimistic UI. In short you've all what Meteor offers here, and it's probably going to be easier to integrate with your existing backend that I assume is Meteor.
Don't include Meteor. Since you're building a chat app, you'll probably need socket.io over a traditional REST API. For sure you can do a mix of both
Use Meteor DDP. (it's kind of like socket.io, but for Meteor. Meteor app use that for all requests to the server) This will include less things that the full Meteor and probably be easier to integrate to your Meteor backend than a REST API / socket.io, and will be some extra work over the full Meteor.
3 How can I implement some sort of unique identifier/user auth for the user and the widget?
This part should probably do some work on the customer website (vs in your iframe) so that you can set cookies on his page, and send that data to your iframe that's gonna talk to your server and identify the user. Wether you use artwells:accounts-guest (that's based on meteor:accounts-base) is going to depend on wether you decide to include Meteor in your iframe.
If you don't have Meteor in your iframe, you can do something like:
handle user creation yourself, by simply doing on your server
.
const token = createToken();
Users.insert({ tokens: [token] });
// send the token back to your iframe
// and set is as a cookie on your customer website
then for each call to your server, on your iframe:
.
let token;
const makeRequest = async (request) => {
token = token || getCookieFromCustomerWebsite();
// pass the token to your HTTP / socket.io / ... request.
// in the header of whatever
return await callServer(token, request);
};
in the server have a middleware that sets the user. Mine looks like:
.
const loginAs = (userId, cb) => {
DDP._CurrentInvocation.withValue(new DDPCommon.MethodInvocation({
isSimulation: false,
userId,
}), cb);
};
// my middleware that run on all API requests for a non Meteor client
export const identifyUserIfPossible = (req, res, next) => {
const token = req.headers.authorization;
if (!token) {
return next();
}
const user = Users.findOne({ tokens: token });
if (!user) {
return next();
}
loginAs(user._id, () => {
next();
// Now Meteor.userId() === user._id from all calls made on that request
// So you can do Meteor.call('someMethod') as you'd do on a full Meteor stack
});
};
Asking your customers to embed your code like this doesn't follow the principles of Security by Design.
From their point of view, you are asking them to embed your prebundled code into their website, exposing their site up to any hidden security risks (inadvertent or deliberately malicious) that exist in your code which would have unrestricted access to their website's DOM, localstorage, etc.
This is why using an iframe is the prefered method to embed third party content in a website, as that content is sandboxed from the rest of it's host site.
Further, following the security principle of 'Least Privilege' they (with your guidance/examples) can set the sandbox attribute on the iframe, and explicitly lockdown via a whitelist the privileges the widget will have.
Loading your widget in an iframe will also give you more flexibility in how it communicates with your servers. This could now be a normal meteor client, using meteor's ddp to communicate with your servers. Your other suggestions are also possible.
User auth/identification depends on the details of your system. This could range from using Meteor Accounts which would give you either password or social auth solutions. Or you could try an anonymous accounts solution such as artwells:accounts-guest.
html5rocks article on sandboxed-iframes