Develop Reference

c# Cefsharp how to make correct sequence of JavaScript actions on the web site

These sequences of actions work with Thread.Sleep, somewhere in 1 second, somewhere in 2 seconds. I think using Thread.Sleep/Task.Delay is not good. Because it can be performed differently on different computers. How do I execute these sequences without using Thread.Sleep?
Or it is OK to using Thread.Sleep/Task.Delay?
private async void ButtonFind_Click(object sender, EventArgs e)
{
//Action1
string jsScript1 = "document.getElementById('story').value=" + '\'' + textFind.Text + '\'';
await chrome.EvaluateScriptAsync(jsScript1);
//Action2
string jsScript2 = "document.querySelector('body > div.wrapper > div.header > div.header44 > div.search_panel > span > form > button').click();";
await chrome.EvaluateScriptAsync(jsScript2);
//Action3
Thread.Sleep(1000); //it is necessary to set exactly 1 seconds
string jsScript3 = "document.getElementsByTagName('a')[2].click();";
await chrome.EvaluateScriptAsync(jsScript3);
//Action4
Thread.Sleep(2000); //it is necessary to set exactly 2 seconds
string jsScript4 = "document.querySelector('#dle-content > div.section > ul > li:nth-child(3)').click();";
await chrome.EvaluateScriptAsync(jsScript4);
}
I tried to use task expectations, but it didn't help me
...
var task4 = chrome.EvaluateScriptAsync(jsScript4);
task4.Wait();
I also tried to use DOM rendering expectations, which didn't help either
string jsScript4 = #"
if( document.readyState !== 'loading' ) {
myInitCode();
} else {
document.addEventListener('DOMContentLoaded', function () {
myInitCode();
});
}
function myInitCode() {
var a = document.querySelector('#dle-content > div.section > ul > li:nth-child(3)').click();
return a;
}
";
chrome.EvaluateScriptAsync(jsScript4);
My addition (21.04.2022)
In third action instead of using Thread.Sleep, im using "While" loop
Here the algorithm is correct, but for some reason, after pressing the application button, the application is hanging
bool test = false;
while(test == false)
{
string myScript = #"
(function(){
var x = document.getElementsByTagName('a')[1].outerText;
return x;
})();
";
var task = chrome.EvaluateScriptAsync(myScript);
task.ContinueWith(x =>
{
if (!x.IsFaulted)
{
var response = x.Result;
if (response.Success == true)
{
var final = response.Result;
if (final.ToString() == textFind.Text)
{
MessageBox.Show("You found the link");
test = true;
}
else
{
MessageBox.Show("You do not found the link");
}
}
}
}, TaskScheduler.FromCurrentSynchronizationContext());
}
My addition (23.04.2022)
string jsScript1 = "document.getElementById('story').value=" + '\'' + textFind.Text + '\'' + ";"
+ #"
Promise.resolve()
.then(() => document.querySelector('body > div.wrapper > div.header > div.header44 > div.search_panel > span > form > button').click())
.then(() => { var target = document.body;
const config = {
childList: true,
attributes: true,
characterData: true,
subtree: true,
attributeFilter: ['id'],
attributeOldValue: true,
characterDataOldValue: true
}
const callback = function(mutations)
{
document.addEventListener('DOMContentLoaded', function(){
if(document.getElementsByTagName('a')[1].innerText=='Troy')
{
alert('I got that link');
}
}, true);
};
const observer = new MutationObserver(callback);
observer.observe(target, config)});
";
var task1 = chrome.EvaluateScriptAsPromiseAsync(jsScript1);
task1.Wait();
Using a MutationObserver wrapped in a promise, using EvaluateScriptAsPromiseAsync to evaluate promise. Also didnt help.
I came to the conclusion that JavaScript does not save the code when clicking on a search button or after going to another page. How do I save the JavaScript code/request and continue it after clicking on a search button or after going to another page?

As your JavaScript causes a navigation you need to wait for the new page to load.
You can use something like the following to wait for the page load.
// create a static class for the extension method
public static Task<LoadUrlAsyncResponse> WaitForLoadAsync(this IWebBrowser browser)
{
var tcs = new TaskCompletionSource<LoadUrlAsyncResponse>(TaskCreationOptions.RunContinuationsAsynchronously);
EventHandler<LoadErrorEventArgs> loadErrorHandler = null;
EventHandler<LoadingStateChangedEventArgs> loadingStateChangeHandler = null;
loadErrorHandler = (sender, args) =>
{
//Actions that trigger a download will raise an aborted error.
//Generally speaking Aborted is safe to ignore
if (args.ErrorCode == CefErrorCode.Aborted)
{
return;
}
//If LoadError was called then we'll remove both our handlers
//as we won't need to capture LoadingStateChanged, we know there
//was an error
browser.LoadError -= loadErrorHandler;
browser.LoadingStateChanged -= loadingStateChangeHandler;
tcs.TrySetResult(new LoadUrlAsyncResponse(args.ErrorCode, -1));
};
loadingStateChangeHandler = (sender, args) =>
{
//Wait for while page to finish loading not just the first frame
if (!args.IsLoading)
{
browser.LoadError -= loadErrorHandler;
browser.LoadingStateChanged -= loadingStateChangeHandler;
var host = args.Browser.GetHost();
var navEntry = host?.GetVisibleNavigationEntry();
int statusCode = navEntry?.HttpStatusCode ?? -1;
//By default 0 is some sort of error, we map that to -1
//so that it's clearer that something failed.
if (statusCode == 0)
{
statusCode = -1;
}
tcs.TrySetResult(new LoadUrlAsyncResponse(statusCode == -1 ? CefErrorCode.Failed : CefErrorCode.None, statusCode));
}
};
browser.LoadingStateChanged += loadingStateChangeHandler;
browser.LoadError += loadErrorHandler;
return tcs.Task;
}
// usage example
private async void ButtonFind_Click(object sender, EventArgs e)
{
//Action1
string jsScript1 = "document.getElementById('story').value=" + '\'' + textFind.Text + '\'';
await chrome.EvaluateScriptAsync(jsScript1);
//Action2
string jsScript2 = "document.querySelector('body > div.wrapper > div.header > div.header44 > div.search_panel > span > form > button').click();";
await Task.WhenAll(chrome.WaitForLoadAsync(),
chrome.EvaluateScriptAsync(jsScript2));
//Action3
string jsScript3 = "document.getElementsByTagName('a')[2].click();";
await Task.WhenAll(chrome.WaitForLoadAsync(),
chrome.EvaluateScriptAsync(jsScript3));
//Action4
string jsScript4 = "document.querySelector('#dle-content > div.section > ul > li:nth-child(3)').click();";
await chrome.EvaluateScriptAsync(jsScript4);
}

You never must work with sleep because time changes between computers and, even in the same computer, a web page may be differ the time required to load.
I work a lot with scraping and IMO the best focus to manage this is working from JavaScript side. You inject/run your JavaScript to fill controls, click buttons...
With this focus, the problem is that navigations make you lose the state. When you navigate to other page, your JavaScript start from scratch. I revolve this sharing data to persist between JavaScript and C# through Bound Object and injecting JavaScript.
For example, you can run action 1, 2 and 3 with a piece of JavaScript code. Before click button, you can use your Bound Object to tell to your C# code that you are going to second page.
When your second page are loaded, you run your JavaScript for your second page (you know the step and can inject the JavaScript code for your 2 page).
In all cases, your JavaScript code must have some mechanism to wait. For example, set a timer to wait until your controls appears. In this way, you can run your JavaScript without wait to the page is fully loaded (sometimes this events are hard to manage).
UPDATE
My scraping library is huge. I'm going to expose pieces that you need to do the work but you need to assemble by yourself.
We create a BoundObject class:
public class BoundObject
{
public BoundObject(IWebBrowser browser)
{
this.Browser = browser;
}
public void OnJavaScriptMessage(string message)
{
this.Browser.OnJavaScriptMessage(message);
}
}
IWebBrowser is an interface of my custom browser, a wrapper to manage all I need. Create a Browser class, like CustomBrowser, for example, implementing this interface.
Create a method to ensure your Bound Object is working:
public void SetBoundObject()
{
// To get events in C# from JavaScript
try
{
var boundObject = new BoundObject();
this._browserInternal.JavascriptObjectRepository.Register(
"bound", boundObject, false, BindingOptions.DefaultBinder);
this.BoundObject = boundObject;
}
catch (ArgumentException ex)
{
if (!ex.ParamName.Identical("bound"))
{
throw;
}
}
}
_browserInternal is the CefSharp browser. You must run that method on each page load, when you navigate. Doing that, you have a window.bound object in JavaScript side with an onJavaScriptMessage function. Then, you can define a function in JavaScript like this:
function sendMessage(msg) {
var json = JSON.stringify(msg);
window.bound.onJavaScriptMessage(json);
return this;
};
You can send now any object to your C# application and manage in your CustomBrowser, on OnJavaScriptMessage method. In that method I manage my custom message protocol, like a typical one in sockets environment or the windows message system and generate a OnMessage that I implement in classes inheriting CustomBrowser.
Send information to JavaScript is trivial using ExecuteScriptAsync of CefSharp browser.
Going further
When I work in an intense scraping job. I create some scripts with classes to manage the entire Web to scrap. I create classes, for example, to do login, navigate to different sections, fill forms... like if I was the owner of the WebSite. Then, when page load, I inject my scripts and I can use my own classes in the remote WebSite making scraping... piece of cake.
My scripts are embedded resources so are into my final executable. In debug, I read them from disk to allow edit+reload+test until my scripts works fine. With the DevTools you can try in the console until you get the desired source. Then you add into your JavaScripts classes and reload.
You can add simple JavaScript with ExecuteScriptAsync, but with large files appears problems escaping quotes...
So you need insert an entire script file. To do that, implement ISchemeHandlerFactory to create and return an IResourceHandler. That resource handler must have a ProcessRequestAsync in which you receive a request.Url that you can use to locale your scripts:
this.ResponseLength = stream.Length;
this.MimeType = GetMimeType(fileExtension);
this.StatusCode = (int)HttpStatusCode.OK;
this.Stream = stream;
callback.Continue();
return true;
stream maybe a MemoryStream in which you write the content of your script file.

How to make this run only once?

I'm trying to get it where If someone clicks the button it will update the database but what happens if I enter 50 then it will keep running it and I have a tracking board that sums everything up so it overloads my server and makes the total in the 1000's when its normally just over 100.
I've tried a document ready function, I've had on and one. ('click') and it keeps running multiple times
$('#update_new_used-counter').one('click', function (event) {
event.preventDefault();
let updated_new_counter = $('#new_sold-update').val().trim();
let updated_used_counter = $('#used_sold-update').val().trim();
trackingBoardRef.on("value", function(snapshot) {
let current_new_counter = snapshot.val().new;
let current_used_counter = snapshot.val().used;
if (updated_new_counter == '') {
trackingBoardRef.update({
new: current_new_counter,
});
} else {
trackingBoardRef.update({
new: updated_new_counter,
})
};
if (updated_used_counter == '') {
trackingBoardRef.update({
used: current_used_counter,
});
} else {
trackingBoardRef.update({
used: updated_used_counter,
})
};
console.log(snapshot.val().new);
console.log(snapshot.val().used);
});
});
That's what I have now and it just keeps running multiple times until firebase says I had to many requests and stops it. I just want it to update once

When you call:
trackingBoardRef.on("value", function(snapshot) {
You attach a listener to the data in trackingBoardRef that will be triggered right away with the current value, and then subsequently whenever the data under trackingBoardRef changes. And since you're changing data under trackingBoardRef in your code, you're creating an infinite loop.
If you only want to read the data once, you can use the aptly named once method:
trackingBoardRef.once("value", function(snapshot) {
...
Note that if you're update the value under trackingBoardRef based on its current value, you really should use a transaction to prevent users overwriting each other's changes.

Why can’t I catch certain exceptions in a MarkLogic request?

I have some code that exercises the “invalid values” setting on an element range index. In this case, I have configured a dateTime element range index on the onDate element in my database (which will apply to both XML elements and JSON properties). I’ve set that index to reject invalid values. This setting means if I try to set the value of an onDate element and it is not castable to a dateTime or is null (literal null in JSON or xsi:nil="true" in XML), my update will fail. (The opposite behavior is to completely ignore invalid values.)
I tried the following code in Server-Side JavaScript in MarkLogic 8.0-4:
'use strict';
declareUpdate();
var errors = [];
var inputs = {
'/37107-valid.json': (new Date()).toISOString(),
'/37107-invalid.json': 'asdf', // Should throw an error
'/37107-null.json': null
};
for(var uri in inputs) {
try {
xdmp.documentInsert(
uri,
{ 'onDate': inputs[uri] },
xdmp.defaultPermissions(),
['37107'] // Collections
);
} catch(err) {
errors.push(err);
}
}
errors.length;
I would have expected my request to succeed and to end up with 1 === errors.length, because only the second insert would have failed because 'asdf' is not castable as a dateTime and it is not null. However, instead I get an XDMP-RANGEINDEX error and my transaction fails. Why doesn’t my try/catch work here?

The issue is how MarkLogic processes update transactions. Rather than actually changing the data with each xdmp.docuentInsert(…) call, MarkLogic queues up all of the updates and applies them atomically at the end of the request. (This is also why you can’t see database updates within the same transaction.) Thus, the error isn’t being thrown until after the loop has executed and the database tries to commit the queued transactions. This behavior is the same in XQuery (slightly simplified):
let $uris := (
'/37107-valid.xml',
'/37107-invalid.xml',
'/37107-null.xml'
)
let $docs := (
<onDate>{fn:current-dateTime()}</onDate>,
<onDate>asdf</onDate>,
<onDate xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"/>
)
return
for $uri at $i in $uris
return
try {
xdmp:document-insert($uri, $docs[$i], (), ('37107'))
} catch($err) {
xdmp:log($err)
}
In order to catch the errors synchronously, you’d need to put each update into its own transaction. In general, this approach will be much slower and resource intensive than MarkLogic’s default transaction handling. However, it’s illustrative here to demonstrate what’s happening under the covers and can come in handy for specific use cases, like this one.
In the example below, I use xdmp.invokeFunction() to “call” a function in a separate transaction from the parent request. (First-class functions for the win!) This allows the updates to be fully applied (or rolled back with an error) and the calling module to see the updates (or errors). I’ve wrapped the low-level xdmp.invokeFunction() in my own applyAs() function to provide some niceties, like correctly passing function arguments to the curried function.
'use strict';
var errors = [];
var inputs = {
'/37107-valid.json': (new Date()).toISOString(),
'/37107-invalid.json': 'asdf',
'/37107-null.json': null
};
var insert = applyAs(
function(uri, value) {
return xdmp.documentInsert(
uri,
{ 'onDate': inputs[uri] },
xdmp.defaultPermissions(),
['37107']
);
},
{ isolation: 'different-transaction', transactionMode: 'update' },
'one'
);
for(var uri in inputs) {
try {
insert(uri, inputs[uri]);
} catch(err) {
errors.push(err);
}
}
errors.length; // Correctly returns 1
// <https://gist.github.com/jmakeig/0a331823ad9a458167f6>
function applyAs(fct, options, returnType /* 'many', 'one', 'iterable' (default) */) {
options = options || {};
return function() {
var params = Array.prototype.slice.call(arguments);
// Curry the function to include the params by closure.
// xdmp.invokeFunction requires that invoked functions have
// an arity of zero.
var f = (function() {
return fct.apply(null, params);
}).bind(this);
// Allow passing in user name, rather than id
if(options.user) { options.userId = xdmp.user(options.user); delete options.user; }
// Allow the functions themselves to declare their transaction mode
if(fct.transactionMode && !(options.transactionMode)) { options.transactionMode = fct.transactionMode; }
var result = xdmp.invokeFunction(f, options); // xdmp.invokeFunction returns a ValueIterator
switch(returnType) {
case 'one':
// return fn.head(result); // 8.0-5
return result.next().value;
case 'many':
return result.toArray();
case 'iterable':
default:
return result;
}
}
}

Firebase data gets overwritten ! sought guidance

I am building an Angular App with Firebase.
My intention is to create an object (say Rooms) at the root with 3 child objects (say Room1, Room2 & Room3) . Also, I am trying to create a logic that would check if the Rooms object is there - it wont create it again.
My code was :
var ref = new Firebase(firebaseURL);
ref.child('Rooms').once('value', function (snapshot){
if(snapshot.numChildren() == 0){
// Create Room within a loop
ref.child('Rooms').child(i).set(roomObj);
}else if(snapshot.numChildren() > 0){
// do not create
}
}
But when the code runs - it always enters into the if block !! And creates the child Rooms.
What is my mistake in the code ??

Most likely the value event will be triggered again with the value you expect.
Your solution is to run the code in a transaction.
var ref = new Firebase(firebaseURL);
ref.child('Rooms').transaction(function (data){
if(!data){
var rooms = {};
for (var roomNum=0; roomNum < 3; roomNum++) {
rooms['room'+roomNum] = { name: 'Room '+roomNum };
}
return rooms
}
}
So if the rooms don't exist yet, the above code creates them. If they already exist, the code does nothing (not returning a value, leaves the data unmodified).
Be sure to read the Firebase documentation for transaction.

Breeze Partial initializer

I have a Single Page Application that is working pretty well so far but I have run into an issue I am unable to figure out. I am using breeze to populate a list of projects to be displayed in a table. There is way more info than what I actually need so I am doing a projection on the data. I want to add a knockout computed onto the entity. So to accomplish this I registered and entity constructor like so...
metadataStore.registerEntityTypeCtor(entityNames.project, function () { this.isPartial = false; }, initializeProject);
The initializeProject function uses some of the values in the project to determine what the values should be for the computed. For example if the Project.Type == "P" then the rowClass should = "Red".
The problem I am having is that all the properties of Project are null except for the ProjNum which happens to be the key. I believe the issue is because I am doing the projection because I have registered other initializers for other types and they work just fine. Is there a way to make this work?
EDIT: I thought I would just add a little more detail for clarification. The values of all the properties are set to knockout observables, when I interrogate the properties using the javascript debugger in Chrome the _latestValue of any of the properties is null. The only property that is set is the ProjNum which is also the entity key.
EDIT2: Here is the client side code that does the projection
var getProjectPartials = function (projectObservable, username, forceRemote) {
var p1 = new breeze.Predicate("ProjManager", "==", username);
var p2 = new breeze.Predicate("ApprovalStatus", "!=", "X");
var p3 = new breeze.Predicate("ApprovalStatus", "!=", "C");
var select = 'ProjNum,Title,Type,ApprovalStatus,CurrentStep,StartDate,ProjTargetDate,CurTargDate';
var isQaUser = cookies.getCookie("IsQaUser");
if (isQaUser == "True") {
p1 = new breeze.Predicate("QAManager", "==", username);
select = select + ',QAManager';
} else {
select = select + ',ProjManager';
}
var query = entityQuery
.from('Projects')
.where(p1.and(p2).and(p3))
.select(select);
if (!forceRemote) {
var p = getLocal(query);
if (p.length > 1) {
projectObservable(p);
return Q.resolve();
}
}
return manager.executeQuery(query).then(querySucceeded).fail(queryFailed);
function querySucceeded(data) {
var list = partialMapper.mapDtosToEntities(
manager,
data.results,
model.entityNames.project,
'ProjNum'
);
if (projectObservable) {
projectObservable(list);
}
log('Retrieved projects using breeze', data, true);
}
};
and the code for the partialMapper.mapDtosToEntities function.
var defaultExtension = { isPartial: true };
function mapDtosToEntities(manager,dtos,entityName,keyName,extendWith) {
return dtos.map(dtoToEntityMapper);
function dtoToEntityMapper(dto) {
var keyValue = dto[keyName];
var entity = manager.getEntityByKey(entityName, keyValue);
if (!entity) {
extendWith = $.extend({}, extendWith || defaultExtension);
extendWith[keyName] = keyValue;
entity = manager.createEntity(entityName, extendWith);
}
mapToEntity(entity, dto);
entity.entityAspect.setUnchanged();
return entity;
}
function mapToEntity(entity, dto) {
for (var prop in dto) {
if (dto.hasOwnProperty(prop)) {
entity[prop](dto[prop]);
}
}
return entity;
}
}
EDIT3: Looks like it was my mistake. I found the error when I looked closer at initializeProject. Below is what the function looked like before i fixed it.
function initializeProject(project) {
project.rowClass = ko.computed(function() {
if (project.Type == "R") {
return "project-list-item info";
} else if (project.Type == "P") {
return "project-list-item error";
}
return "project-list-item";
});
}
the issue was with project.Type I should have used project.Type() since it is an observable. It is a silly mistake that I have made too many times since starting this project.
EDIT4: Inside initializeProject some parts are working and others aren't. When I try to access project.ProjTargetDate() I get null, same with project.StartDate(). Because of the Null value I get an error thrown from the moment library as I am working with these dates to determine when a project is late. I tried removing the select from the client query and the call to the partial entity mapper and when I did that everything worked fine.

You seem to be getting closer. I think a few more guard clauses in your initializeProject method would help and, when working with Knockout, one is constantly battling the issue of parentheses.
Btw, I highly recommend the Knockout Context Debugger plugin for Chrome for diagnosing binding problems.
Try toType()
You're working very hard with your DTO mapping, following along with John's code from his course. Since then there's a new way to get projection data into an entity: add toType(...) to the end of the query like this:
var query = entityQuery
.from('Projects')
.where(p1.and(p2).and(p3))
.select(select)
.toType('Project'); // cast to Project
It won't solve everything but you may be able to do away with the dto mapping.
Consider DTOs on the server
I should have pointed this out first. If you're always cutting this data down to size, why not define the client-facing model to suit your client. Create DTO classes of the right shape(s) and project into them on the server before sending data over the wire.
You can also build metadata to match those DTOs so that Project on the client has exactly the properties it should have there ... and no more.
I'm writing about this now. Should have a page on it in a week or so.