Develop Reference

How to do http request to get the whole source page when part of html loaded by javascript?

I wait to get the html web page from https://www.collinsdictionary.com/dictionary/english/supremacy, but part of the html file is loaded by javascript. When I use HTTP.jl to get the web page with HTTP.request(), I only get part of the html file that loaded before the javascript been run, so the web page I get is different to the web page I got from Chrome. How can I get the web page as same as Chrome get? Do I have to use WebDriver.jl with is a a wrapper around Selenium WebDriver's python bindings?
part of my source:
function get_page(w::word)::Bool
response = nothing
try
response = HTTP.request("GET", "https://www.collinsdictionary.com/dictionary/$(dictionary)/$(w.org_word)",
connect_timeout=connect_timeout, readtimeout=readtimeout, retries=retries, redirect=true,proxy=proxy)
catch e
push!(w.err_log, [get_page_http_err, string(e)])
return falses
end
open("./assets/org_page.html", "w") do f
write(f, String(response.body))
end
return true
end
dictionary and w.org_word are both String, the function is in a module.

What you want is impossible to achieve with just HTTP.jl. Running the Javascript part of the page is fundamentally different -- you need a Javascript engine to do so, which is nothing simple.
And this is not a unique weakness of Julia's HTTP:
Python requests.get(url) returning javascript code instead of the page html
(recently the standard library request in python seems to added Javascript rendering ability)

How the get the whole source html code of a webpage that is generated by Javascript using Java / Webdriver?

I am a newbie in programming and I have a task here I need to solve. I am trying to get the html source code of a webpage using Java / Webdriver method getPageSource(). Problem is, that page is somehow generated, probably by javascript, so the result I get is html code containing just page skeleton - a table that is empty, not filled by data. But, there is tag like <script type="text/javascript" src="/x/js/main.c0e805a3.js"></script> in the very bottom of that html code.
The question is, how can I force Webdriver to run that Javascript and give me the result - the whole source html with data. I already tried to use this (js.executeScript("window.location = '/x/js/main.c0e805a3.js'");) before calling getPageSource() but not successful.
Any help will be appreciated, thanks!

There are quite a few setups, now, that can run the Java-Script on a web-page. The most well known, I think, is likely Selenium since I think it has been around for a while. Others include karate, Puppeteer, and even an old tool called Rhino. Puppeteer is a Google, Inc. project that uses Java-Script (server-side Java-Script, called Node.js. They don't like us comparing, contrasting libraries here.
I haven't had the time to engage Selenium, yet, but I write HTML parser, search and update code all the time. If your only goal is to load a page whose contents are dynamically "filled in by AJAX calls" - and what I mean by that, you only want the contents of an HTML that would normally see when you visit the sites web-page, and you are not concerned with button presses then the one I have been using for that is called Splash This tool does have the ability to let you invoke Java-Script, but if all you want to do is see the JS on a page dynamically load the table, then, literally, all you have to do is start-the tool, and add one line to your program.
On Google Cloud Platform, these 2 lines will start a Splash Proxy Server. If you are writing your code on AWS (Amazon) or Azure (Microsoft), it would likely be similar. If you are running your code in an office on the local machine, you would have to research how to start it.
Install Docker. Make sure Docker version >= 17 is installed.
Pull the image:
$ sudo docker pull scrapinghub/splash
Start the container:
$ sudo docker run -it -p 8050:8050 --rm scrapinghub/splash
Then, in your code, all you have to do is the following:
// If your original code looked like this:
URL url = new URL("https://en.wikipedia.org/wiki/Christopher_Columbus");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", USER_AGENT);
return new BufferedReader(new InputStreamReader(con.getInputStream()));
Change the first line of code in this example to this, and (theoretically), and dynamically loaded HTML tables that are completed with the onload page events will be automatically loaded before returning the HTML page.
// Add this line to your methods
String splashProxy = "http://localhost:8050/render.html?url=";
URL url = new URL(splashProxy + "https://en.wikipedia.org/wiki/Christopher_Columbus");
For most web-sites, any initial tables that are filled by JS/jQuery/AJAX will be filled in. If you are willing to learn teh Lua Programming Language, you can also start invoking the methods there. It has been pretty convenient for my purposes, since I am not writing web-page testing code (code that simulates user button presses). If that is what you are doing, Selenium is likely worth spending time learning / studying the A.P.I.

Run Java method in Javascript in Freemarker template

I would like to run method over my Java object in Freemaker template but in Javascript. Could be also directly in Freemarker but I need to run it on click.
I have issue that following method doesnt run:
actual_id.setActual_id(variable) ;
And I have following Java code:
Setting actual_id= new Setting("-");
Map<String, Object> data = new HashMap<>();
data.put("items", items);
data.put("actual_id", actual_id);
public Setting(String actual_id) {
this.actual_id = actual_id;
}
public String getActual_id() {
return actual_id;
}
public void setActual_id(String actual_id) {
this.actual_id = actual_id;
}
This is my Freemaker template:
<script src="https://code.jquery.com/jquery-1.10.2.js"></script>
<script>function myFunction(variable) {
alert(variable);
actual_id.setActual_id(variable) ;
location.reload();}
</script>
<#list items as item>
<p>${item.name}: ${item.id} <button type="button" id="${item.id}" onclick=myFunction("${item.id}") >Details</button>
</#list>
EDIT1:
I am also trying something like this:
onclick="${actual_id.setActual_id(item.id)}"
but cannot trigger activity from onclick.
EDIT2:
I already almost found solution. Following code executes Java method, I just need to figure out how to quote "variable" to load javasript value into it.
function myFunction(variable) {
alert(variable);
var idd ="${actual_id.setActual_id(variable)}";//here variable value needed
alert(idd);
location.reload();
}

Here is a quote from the Apache FreeMarker Project front-page:
(https://freemarker.apache.org/)
What is Apache FreeMarker™?
Apache FreeMarker™ is a template engine: a Java library to generate
text output (HTML web pages, e-mails, configuration files, source
code, etc.) based on templates and changing data. Templates are
written in the FreeMarker Template Language (FTL), which is a simple,
specialized language (not a full-blown programming language like PHP).
Usually, a general-purpose programming language (like Java) is used to
prepare the data (issue database queries, do business calculations).
Then, Apache FreeMarker displays that prepared data using templates.
In the template you are focusing on how to present the data, and
outside the template you are focusing on what data to present.
Figure [Photo/Image Not Posted]
This approach is often referred to as the MVC (Model View Controller)
pattern, and is particularly popular for dynamic web pages. It helps
in separating web page designers (HTML authors) from developers (Java
programmers usually). Designers won't face complicated logic in
templates, and can change the appearance of a page without programmers
having to change or recompile code.
While FreeMarker was originally created for generating HTML pages in
MVC web application frameworks, ** it isn't bound to servlets or HTML
or anything web-related.** It's used in non-web application
environments as well.
https://freemarker.apache.org/
I, myself, program Java & JavaScript web-servers on Google Cloud Server all day long. The only way to make a JavaScript function talk to a Java Function is through an HTTP GET / POST call to a Java-Servlet or, also, an old-school JSP Page. Though it says (explicity) right on the top-level domain page of the website that "Freemarker is not bound to Servlets" - that actually means the software classes / package does not have to run inside of a web-environment at all - perhaps on your desktop computer without a web-browser.
What I do know with an extremely high degree of certainty is that communication between the client (on a web-browser) and a server (a web-server) is always done through HTTP GET / POST requests. JSON, AJAX are common for higher communicating large amounts of data. If you expect a JavaScript method to make a call to a Java Class on the back-end, you will need to include a Servlet or JSP class - and the whole 9 yards to boot.
NOTE: I have not used Apache FreeMarker, but I program Java/JavaScript all day long. Judge accordingly! According to the Apache web-site, FreeMarker is of assistance in "programatically or automatically generating the HTML for pages" (which is what C# is good at) - which is actually something I do for my web-site, often, but (alas!) I don't use Apache's product. What that means is FreeMarker can help the generating of HTML more efficiently using Java Classes on the back-end server side ...
But the rules of how Java and Java-Script communicate have not
changed...
Long story short - you must include JavaScript calls such as:
calling a java servlet from javascript
How to send request from javascript to servlet?
How to call servlet from javascript

Android: modify JS loaded by a local static html based on selection from list in previous activity

Still very much a beginner to Android and Java, so thanks in advance for your patience!
I am trying to find a means of passing a file path (which could be in the form of a simple string or integer) from an activity (in Java) to a WebView (which is rendering a static local html file that is running JS). The issue is complicated by the fact that I am using an AR SDK (Wikitude) and the port to the SDK uses their own customised WebView(which also renders a CameraSurfaceView simultaneously) - controlling their suite is then done in a JS file loaded by the html file.
Any solution is welcome, I am new to Android so don't even know where to begin on this one. In simplest terms, I want to take a chosen option from one activity and use that information to tell my JS file what asset to load/render. If this were a web app you could use some templating to load a dynamic variable into your html (e.g. Using erb in ruby or moustache in JS).
If there is no equivalent for templating Java into a JS file, my current best guess is to write a JSON object using JsonWriter in the activity and subsequently load that file in the JS, if this is indeed the best solution, I am struggling to navigate the internal / external storage of android relative to the assets folder of my app - can anyone shed some light on how I would do that?
I've tried to keep this as general as possible, code can be provided on request.

you can pass chosen option from one activity when you are starting second activity with following code.
//Code to start another activity
Intent intent = new Intent(MainActivity.this, Second_Activity.class);
pass the value with intent
intent.putExtra("choice1", "choice1");
intent.putExtra("choice1", "choice1");
startActivity(intent);
now in second activity onCreate() you can receive these value from intent using following code
String choice1 = getIntent().getStringExtra("choice1");

Passing Value from C++ to Javascript

i have a c++ file which reads values from a sensor and I want to display those values on a website dynamically. So Im looking for a way to pass these values(integers) from my cpp file to an javascript which displays them on the site.
My first, simple try was to write the values into a js file as variables every second from my cpp script. The Js then uses this file as a source and displays its variables on the site:
cpp:
fprintf(file, "var mx=%d, my=%d, mz=%d, ax=%d, ay=%d, az=%d, gx=%d, gy=%d, gz=%d;\n",
imu.raw_m[0], imu.raw_m[1], imu.raw_m[2], // M = Magnetometer
imu.raw_a[0], imu.raw_a[1], imu.raw_a[2], // A = Accelerometer
imu.raw_g[0], imu.raw_g[1], imu.raw_g[2] // G = Gyroscope
);
html/js:
<script src="./imu.js" type="text/javascript"></script>
The Problem now is of course, that I need to refresh the page all the time, because the imu.js file is cached by the website.
I'd rather have a way to directly pass to integers from the cpp file to the js script. I read something about json or Googles V8 script. But I'd like to hear your suggestions first.
By the way, Im running this on a raspi, if this is important.
Thanks for your help
EDIT:
I'm goning to try it with a mysql database, in which my cpp file writes the data from the sensor with Connector/c++ from http://dev.mysql.com/doc/connector-cpp/en/ and my website reads them.

You could compile your C++ code into a Node.js plugin, you can then register a JavaScript function with your plugin which the C++ calls when it updates the value. That way you can pass values directly from C++ into Javascript in a managed and controlled way.
Node.js has the added benefit of being able to host your webpage and do all the Websocket and HTTP stuff that can be a pain in C++.

You do not have to refresh if your script is smart about how to access the data file! In case you do have a webserver at hand: Take care that your data file is accessible by your webserver and then let your script request the file via ajax (link to w3schools)

I'm doing something similar on a BeagleBone Black. With websocketd you can turn pretty much any program into a websocket endpoint and then send data via stdin and stdout commands. This would be a particularly good solution for you since websockets are designed to handle information that's constantly changing.