I've started using workers for my sites. The purpose of workers on my sites:
mostly inject some code(html/js/css) in different locations of HTML page.
It can be some config data, or some legal text and etc.
So what I do now, is create a config in KV for each website and based on user country/language injecting above html/js and etc.
Below is a Store Class (Singleton pattern), that holds all the info from config, but doesn't work in workers, by doesn't work I mean, after first request, the data is persistent, and after some time it gets updated:
For example 1st request URL: /about
On Second request URL: /es/about/
By output console.log(event.request) will show /es/about , but Store.request outputs: /about/
any workaround for this, to force refresh of data, I thought becuase i don't do it in constructor, but by calling custom method should do the trick but, it doesn't.?
Below is some code example.
import { Store } from "#helpers/store";
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
});
//HTML Rewriter Class
class Head {
element(el) {
el.append(`
<script id="config">
var config = ${Store.export()};
</script>`, {
html: true
});
}
}
async function handleRequest(request) {
let store = await Store.make(request);
const response = await fetch(request);
let html = new HTMLRewriter();
html.on("head", new Head());
return html.transform(response);
}
//src/helpers/store.js
class Store {
constructor(){
this._request = null
this._config = {}
this._url = null
}
async make(request){
let config = {}
this._request = request;
const domain = this.url.hostname.replace(/www\./g, "");
const country = request.headers.get('cf-ipcountry')
const website = await WEBSITES.get(domain, "json"); //WEBSITES is KV namespace
const { license, lang } = website;
this._config = {
country,
domain,
license,
lang
};
return this;
}
export(){
return JSON.stringify(this._config)
}
get request(){
return this._request;
}
get url(){
if(!this._url){
this._url = new URL(this.request.url)
}
return this._url;
}
}
export default new Store()
A single instance of your Worker may handle multiple requests, including concurrent requests. In your code, you are using a singleton instance of Store to store metadata about the current request. But if multiple requests are handled concurrently, then the second request will overwrite the content of Store before the first request completes. This may cause the first request to render its HTML using metadata from the second request.
It seems like the use of a singleton pattern isn't what you want here. Based on the code, it looks like you really want to create a separate instance of Store for every request.
2 issues come to mind:
You are creating a new HTMLRewriter for each call to the worker. This will make for some concurrency issues. The instantiation of the rewriter should be done outside the handleRequst method. For example right after the import statement.
You are importing the Store class and never instantiating it but using its methods like they are static(which they aren't). This will also give you concurrency issues.
Related
I want to check if a container exists and if not, initialize it. I was hoping for something like the following:
const { endpoint, key, databaseId } = config;
const containerName = "container1"
const client = new CosmosClient({ endpoint ,key});
const containerDefinition = getContainerDefinition(containerName);
const db = await createDatabase(client, databaseId);
if (!db.containers.contains(containerName)
{
// Do something
}
The reason I'm not using "createIfNotExists" is because I would need to make a 2nd call to check if the container returned is populated with items or not. The container I'm creating is going to hold settings data which will be static once the container is initially created. This settings check is going to happen per request so I'd like to minimize the database calls and operations if possible.
I tried doing something like:
try
{
db.container(containerName).read();
}
catch(err)
{
if(err.message.contains("Resource Not Found"))
{
// do something
}
}
But that doesn't seem like the right way to do it.
Any help would be appreciated!
I'm not quite clear on why you would need to do this since typically you only need to do this sort of thing once for the life of your application instance. But I would not recommend doing it this way.
When you query Cosmos to test the existence of a database, container, etc., this hits the master partition for the account. The master partition is kind of like a tiny Cosmos database with all of your account meta data in it.
This master partition is allocated a small amount of the RU/s that manage the metadata operations. So if you app is designed to make these types of calls for every single request, it's quite likely you will get rate limited in your application.
If there is some way you can design this such that it doesn't have to query for the existence of a container then I would pursue that instead.
Interesting question. So i think you have few options
Just call const { container } = await database.containers.createIfNotExists({ id: "Container" }); it will be fast probably few milliseconds, since I went via code at looks like it will always try to read from cosmos :( If you want to still check if container exists sdk has methods(But again no real benefits ):
const iterator = database.containers.readAll();
const { resources: containersList } = await iterator.fetchAll();
Create singleton and first time just initialise all your containers so next time you dont call it, sure if you scale each instance will do the same
My favourite, use terraform/armtemplates/bicep to spin up infrastructure so you code wont need to handle that
You can try this code:
async function check_container_exist(databaseId,containerId) {
let exist = false;
const querySpec = {
query: "SELECT * FROM root r WHERE r.id = #container",
parameters: [
{name: "#container", value: containerId}
]
};
const response = await client.database(databaseId).containers.query(querySpec).fetchNext();
if(response.resources[0]){
exist = true;
}
return exist;
}
I wanted to swap a profile picture of a user. For this, I have to check the database to see if a picture has already been saved, if so, it should be deleted. Then the new one should be saved and entered into the database.
Here is a simplified (pseudo) code of that:
async function changePic(user, file) {
// remove old pic
if (await database.hasPic(user)) {
let oldPath = await database.getPicOfUser(user);
filesystem.remove(oldPath);
}
// save new pic
let path = "some/new/generated/path.png";
file = await Image.modify(file);
await Promise.all([
filesystem.save(path, file),
database.saveThatUserHasNewPic(user, path)
]);
return "I'm done!";
}
I ran into the following problem with it:
If the user calls the API twice in a short time, serious errors occur. The database queries and the functions in between are asynchronous, causing that the changes of the first API call weren't applied when the second API checks for a profile pic to delete. So I'm left with a filesystem.remove request for an already unexisting file and an unremoved image in the filesystem.
I would like to safely handle that situation by synchronizing this critical section of code. I don't want to reject requests only because the server hasn't finished the previous one and I also want to synchronize it for each user, so users aren't bothered by the actions of other users.
Is there a clean way to achieve this in JavaScript? Some sort of monitor like you know it from Java would be nice.
You could use a library like p-limit to control your concurrency. Use a map to track the active/pending requests for each user. Use their ID (which I assume exists) as the key and the limit instance as the value:
const pLimit = require('p-limit');
const limits = new Map();
function changePic(user, file) {
async function impl(user, file) {
// your implementation from above
}
const { id } = user // or similar to distinguish them
if (!limits.has(id)) {
limits.set(id, pLimit(1)); // only one active request per user
}
const limit = limits.get(id);
return limit(impl, user, file); // schedule impl for execution
}
// TODO clean up limits to prevent memory leak?
Hello I am trying to access an Httprequest triggerd by a react component from my javascript code in order to test the url?
can anybody help please ?
Screenshot of the httprequest I want to access
Here is an example of the unit test I'am running, I want to add an other unit test that checks if the httprequest is called correctly.
.add('Search with "Occasion" keyword', () => {
const result = search('Iphone Occasion');
specs(() =>
describe('SEO Navigation Links', () => {
it('Should not contain "Occasion" keyword', () => {
const searchValue = result.find(Search).node.state.value.toLowerCase();
const contains = searchValue.includes('occasion');
expect(contains).toBeTruthy();
});
}),
);
return result;
});
The best i can recommend is to "monkey patch" the fetch function (if it uses that)
const realFetch = fetch;
fetch = (...args) => realFetch(...args).then(doStuff);
It creates a "middleware", and when the website tries to call the fetch function, it will call yours
Make sure you make a copy of the original function to avoid infinite recursion
If you install a Service worker, you can run some code client-side on all the requests your page makes. I'm not sure what you need to do in order to test the code you are talking about, but a Service Worker could report the request back to your own test code on the page, or respond with whatever content you want, or modify the server's response.
I am using Restangular to handle my token/header authentication in a single page Angular web application.
Using addFullRequestInterceptor, I set the correct headers for each outgoing REST API call, using a personal key for encrypting data.
Restangular
.setBaseUrl(CONSTANTS.API_URL)
.setRequestSuffix('.json')
.setDefaultHeaders({'X-MyApp-ApiKey': CONSTANTS.API_KEY})
.addFullRequestInterceptor(requestInterceptor)
.addErrorInterceptor(errorInterceptor);
function requestInterceptor(element, operation, route, url, headers, params, httpConfig) {
var timeStamp = Helpers.generateTimestamp(),
//Condensed code for illustration purposes
authSign = Helpers.generateAuthenticationHash(hashIngredients, key, token),
allHeaders = angular.extend(headers, {
'X-MyApp-Timestamp': timeStamp,
'Authentication': authSign
});
return {
headers: allHeaders
}
}
Works great. There is one exception I need though: For a new visitor that has not logged in yet, a generic key/token pair is requested via REST. This key/token pair is used in the headers of the login authentication call.
So for this call, I create a separate Restangular sub-configuration. In this configuration I want to override the requestInterceptor. But this seems to be ignored (i.e. the original interceptor is still called). It doesn't matter if I pass null or a function that returns an empty object.
var specialRestInst = Restangular.withConfig(function(RestangularConfigurer) {
RestangularConfigurer.addFullRequestInterceptor(function() {return {}});
}),
timeStamp = Helpers.generateTimestamp(),
header = {'X-MyApp-Timestamp': timeStamp};
specialRestInst.one('initialise').get({id: 'app'}, header)
So as documented by Restangular, withConfig takes the base confuration and extends it. I would like to know how to removeFullRequestInterceptor (this function does not exist), override it, or something like that.
I would take a different approach and try to pass a flag to the interceptor. If the flag exists then the authSign is excluded. You can do this using withHttpConfig. It's better to exclude on special cases then to always having to tell the interceptor to include the authSign.
So you would update the interceptor like this.
function requestInterceptor(element, operation, route, url, headers, params, httpConfig) {
var timeStamp = Helpers.generateTimestamp();
var allHeaders = {'X-MyApp-Timestamp': timeStamp};
if(!httpConfig.excludeAuth) {
//Condensed code for illustration purposes
var authSign = Helpers.generateAuthenticationHash(hashIngredients, key, token);
allHeaders['Authentication'] = authSign;
}
return angular.extend(headers, allHeaders);
}
When you need to exclude the authSign you would use restangular like this.
specialRestInst.withHttpConfig({excludeAuth: true}).get({id: 'app'});
You should be able to add any values to http config you want as a long as they aren't already used.
I'm not sure if this will work as expected, but I can't see why it wouldn't work.
This pertains to cookies set inside a script (maybe inside a script tag).
System.Windows.Forms.HtmlDocument executes those scripts and the cookies set (like document.cookie=etc...) can be retrieved through its Cookies property.
I assume HtmlAgilityPack.HtmlDocument doesn't do this (execution). I wonder if there is an easy way to emulate the System.Windows.Forms.HtmlDocument capabilities (the cookies part).
Anyone?
When I need to use Cookies and HtmlAgilityPack together, or just create custom requests (for example, set the User-Agent property, etc), here is what I do:
Create a class that encapsulates the request/response. Let's call this class WebQuery
Have a private CookieCollection (in your case public) property inside that class
Create a method inside the class that does manually the request. The signature could be:
...
public HtmlAgilityPack.HtmlDocument GetSource(string url);
What do we need to do inside this method?
Well, using HttpWebRequest and HttpWebResponse, generate the http request manually (there are several examples of how to do this on Internet), create an instance of a HtmlDocument class using the constructor that receives an stream.
What stream do we have to use? Well, the one returned by:
httpResponse.GetResponseStream();
If you use HttpWebRequest to make the query, you can easily set the CookieContainer property of it to the variable you declared before everytime you access a new page, and that way all cookies set by the sites you access will be properly stored in the CookieContainer variable you declared in your WebQuery class, taking in count you're using only one instance of the WebQuery class.
Hope you find useful this explanation. Take in count that using this, you can do whatever you want, no matter if HtmlAgilityPack supports it or not.
I also worked with Rohit Agarwal's BrowserSession class together with HtmlAgilityPack.
But for me subsequent calls of the "Get-function" didn't work, because every time new cookies have been set.
That's why I added some functions by my own. (My solution is far a way from beeing perfect - it's just a quick and dirty fix) But for me it worked and if you don't want to spent a lot of time in investigating BrowserSession class here is what I did:
The added/modified functions are the following:
class BrowserSession{
private bool _isPost;
private HtmlDocument _htmlDoc;
public CookieContainer cookiePot; //<- This is the new CookieContainer
...
public string Get2(string url)
{
HtmlWeb web = new HtmlWeb();
web.UseCookies = true;
web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest2);
web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse2);
HtmlDocument doc = web.Load(url);
return doc.DocumentNode.InnerHtml;
}
public bool OnPreRequest2(HttpWebRequest request)
{
request.CookieContainer = cookiePot;
return true;
}
protected void OnAfterResponse2(HttpWebRequest request, HttpWebResponse response)
{
//do nothing
}
private void SaveCookiesFrom(HttpWebResponse response)
{
if ((response.Cookies.Count > 0))
{
if (Cookies == null)
{
Cookies = new CookieCollection();
}
Cookies.Add(response.Cookies);
cookiePot.Add(Cookies); //-> add the Cookies to the cookiePot
}
}
What it does: It basically saves the cookies from the initial "Post-Response" and adds the same CookieContainer to the request called later. I do not fully understand why it was not working in the initial version because it somehow does the same in the AddCookiesTo-function. (if (Cookies != null && Cookies.Count > 0) request.CookieContainer.Add(Cookies);)
Anyhow, with these added functions it should work fine now.
It can be used like this:
//initial "Login-procedure"
BrowserSession b = new BrowserSession();
b.Get("http://www.blablubb/login.php");
b.FormElements["username"] = "yourusername";
b.FormElements["password"] = "yourpass";
string response = b.Post("http://www.blablubb/login.php");
all subsequent calls should use:
response = b.Get2("http://www.blablubb/secondpageyouwannabrowseto");
response = b.Get2("http://www.blablubb/thirdpageyouwannabrowseto");
...
I hope it helps when you're facing the same problem.