I need to send large string representing a number over a wire to the client written in Java Script. In theory it's a stream of 0s and 1s, like: 011101011101101..., but it can be very, very big (millions of bits in length). My goal, of course, is to minimize necessary processing and sending overhead. I thought about changing the base number of that string, so that it uses HEX or larger radix which would greately reduce the amount of data that has to be sent. JavaScript has built in functions for converting to and from different numbering systems so it looked like the way to go. However, the maximum supported radix is just 36. My calculations are showing that when having a stream of 50 mln bits and using 36 radix, then you still need 1,388,888 characters to be sent - way too much.
My question is - do you know any way that would help to achieve my goal? Some constraints:
Solution must work for streams of arbitrary length
bit streams can be as large as 50mln bits
Good performance should be guaranteed for length around 1-10mln bits. For larger streams it should still work, however it doesn't have to scale up linearly
I would put more emphasis on optimizing the amount of data that has to be sent, not necessarily on reducing the CPU overhead
You could do something like this (the demo has been written with no way of testing it, but it "should work™", here is a fiddle for those who wants to copy/cut/paste: http://fiddle.jshell.net/sywz3aym/2/
Please note the fiddle can't run, i had intended to write a responder but i can't right now i'm afraid.
At the bottom of the javascript entry area i have comment section of how a asp.net responder could look, it uses a "Generic Handler (.ashx)" file if you use visual studio, if you use any other language you will have to use the equivalent options there. You need a request responder which you can customize to return binary data, you need to set the Content-Type to "application/octet-stream" (octet for those who do not know = "group of 8", eg. a byte :))
And here is the javascript + comment as in a glorious wall of text format:
$(document).ready(function () {
var url = 'your handler URL';
var oReq = new XMLHttpRequest();
oReq.open('GET', url, true);
oReq.responseType = 'arraybuffer';
oReq.onload = function (oEvent) {
var buffer = oReq.response;
var yourData = ExtractData(buffer);
}
oReq.send(null);
});
function ExtractData(buffer) {
var dataReader = {
dataView: new DataView(buffer),
readPtr: 0,
littleEndian: (function () {
var buffer = new ArrayBuffer(2);
new DataView(buffer).setInt16(0, 256, true);
return new Int16Array(buffer)[0] === 256;
})(),
setReadPtrOffset: function (byteIndex) {
this.readPtr = byteIndex;
},
nextInt8: function () {
var data = this.dataView.getInt8(this.readPtr);
this.readPtr += 1; // Sizeof int
return data;
},
nextUint8: function () {
var data = this.dataView.getUint8(this.readPtr);
this.readPtr += 1; // Sizeof int8
return data;
},
nextInt32: function () {
var data = this.dataView.getInt32(this.readPtr, this.littleEndian);
this.readPtr += 4; // Sizeof int32
return data;
},
nextUint32: function () {
var data = this.dataView.getUint32(this.readPtr, this.littleEndian);
this.readPtr += 4; // Sizeof uint32
return data;
},
nextFloat32: function () {
var data = this.dataView.getFloat32(this.readPtr, this.littleEndian);
this.readPtr += 4; // Sizeof float
return data;
},
nextFloat64: function () {
var data = this.dataView.getFloat64(this.readPtr, this.littleEndian);
this.readPtr += 8; // Sizeof double
return data;
},
nextUTF8String: function (length) {
var data = String.fromCharCode.apply(null, new Uint8Array(this.dataView.buffer, this.readPtr, length));
this.readPtr += length; // Sizeof int
return data;
},
}
var numberOfInt32ToRead = dataReader.nextInt32(); // First data could be, for example, the number of ints to read.
for(var i = 0; i < numberOfInt32ToRead; i++){
var someInt = dataReader.nextInt32();
// doStuffWithInt(someInt);
}
}
/*
Serverside code looks kind of like this (asp.net/c#):
public class YourVeryNiceDataRequestHandler : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "application/octet-stream"; // <- very important
List<int> data = SomeMethodWithGivesYouData();
context.Response.BinaryWrite(BitConverter.GetBytes((int)data.Count)); // Explicit type casting only to help me get an overview of the data being sent, in this case it has no functional meaning other then to help me debug
foreach(int i in data)
{
context.Response.BinaryWrite(BitConverter.GetBytes((int)i));
}
// You could send structs as well, either with the help of a binary formatter, or you can do like i did here and use BitConverter.GetBytes, be carefull when sending strings/chars since they are by default (usually) in a widechar format (1 char = 2 bytes), since i only use english for this i can convert it to a UTF8 char (1 byte): context.Response.BinaryWrite(System.Text.Encoding.UTF8.GetBytes(new char[] { (motion.Phase == Phases.Concentric ? 'C' : 'E') })); // Phase (as single char)
}
}
*/
I hope this either help you directly, or guide you in the correct direction.
Please note this is VERY dependent on your data types that your server use, and might not fall into your "REST" category, you did however state that you wanted to optimize for stream size and this afaik is the best way to do that, with the exception of adding data compression
Javascript typed arrays "mimics" c-style datatypes, like those used in c, c++, c#, java and so forth
Disclamer:
I have not tried this with data > 1MB, it did however cut the total size of data being sent from server to client from 2MB down to a few 10-100~KB, and it for the amount of data i send/read the execution time were "not humanly measurable" other then the roundtrip time for the extra request, since this is requested AFTER the page is loaded in the browser
Final word of warning: Any manual bit serializing/deserializing should be done with care, since it is very easy to make a mistake forgetting that you have changed something either client or server side, and it will give you un-debuggable garbage if you read or write a byte too much/too little on either end, this is why i add explicit type casting in my server-side code, that way i can open both the server code and the client code side-by-side and match readInt32 to a Write((int)...).
Not a fun job, but it makes things very compact and very fast (normally i would go for readability, but some tasks just have to work faster then readable code).
Typed arrays can't be used in every browser however, but caniuse state that 85% of the internet could use them: http://caniuse.com/#feat=typedarrays
Related
The method I use I need to put +13 and -1 inside the calculation when searching the position of each part of the text (const Before and const After), is there a more reliable and correct way?
const PositionBefore = TextScript.indexOf(Before)+13;
const PositionAfter = TextScript.indexOf(After)-1;
My fear is that for some reason the search text changes and I forget to change the numbers for the calculation and this causes an error in the retrieved text.
The part of text i'm return is date and hour:
2021-08-31 19:12:08
function Clock() {
var sheet = SpreadsheetApp.getActive().getSheetByName('Clock');
var url = 'https://int.soccerway.com/';
const contentText = UrlFetchApp.fetch(url).getContentText();
const $ = Cheerio.load(contentText);
const Before = '"timestamp":"';
const After = '});\n block.registerForCallbacks();';
var ElementSelect = $('script:contains(' + Before + ')');
var TextScript = ElementSelect.html().replace("\n","");
const PositionBefore = TextScript.indexOf(Before)+13;
const PositionAfter = TextScript.indexOf(After)-1;
sheet.getRange(1, 1).setValue(TextScript.substring(PositionBefore, PositionAfter));
}
Example full text colected in var TextScript:
(function() {
var block = new HomeMatchesBlock('block_home_matches_31', 'block_home_matches', {"block_service_id":"home_index_block_homematches","date":"2021-08-31","display":"all","timestamp":"2021-08-31 19:12:08"});
block.registerForCallbacks();
$('block_home_matches_31_1_1').observe('click', function() { block.filterContent({"display":"all"}); }.bind(block));
$('block_home_matches_31_1_2').observe('click', function() { block.filterContent({"display":"now_playing"}); }.bind(block));
block.setAttribute('colspan_left', 2);
block.setAttribute('colspan_right', 2);
TimestampFormatter.format('block_home_matches_31');
})();
There is no way to eliminate the risk of structural changes to the source content.
You can take some steps to minimize the likelihood that you forget to change your code - for example, by removing the need for hard-coded +13 and -1. But there can be other reasons for your code to fail, beyond that.
It's probably more important to make it extremely obvious when your code does fail.
Consider the following sample (which does not use Cheerio, for simplicity):
function demoHandler() {
var url = 'https://int.soccerway.com/';
const contentText = UrlFetchApp.fetch(url).getContentText();
var matchedJsonString = contentText.match(/{.*?"timestamp".*?}/)[0];
if ( matchedJsonString ) {
try {
var json = JSON.parse(matchedJsonString);
} catch(err) {
console.log( err ); // "SyntaxError..."
}
console.log(json.timestamp)
} else {
consle.log( 'Something went terribly wrong...' )
}
}
When you run the above function it prints the following to the console:
2021-08-31 23:18:46
It does this by assuming the key value of "timestamp" is part of a JSON string, starting with { and ending with }.
You can therefore extract this JSON string and convert it to a JavaScript object and then access the timestamp value directly, without needing to handle substrings.
If the JSON is not valid you will get an explicit error similar to this:
[SyntaxError: Unexpected token c in JSON at position 0]
Scraping web page data almost always has these types of risk: Your code can be brittle and break easily if the source structure changes without warning. Just try to make suc changes as noticeable as possible. In your case, write the errors to your spreadsheet and make it really obvious (red, bold, etc.).
And make good use of try...catch statements. See: try...catch
Hey guys I am trying to make a request to a JSON page, grabbing the data, then displaying it to my console but it is giving me "undefined". Why is that?
Here is the code and then the JSON page will be posted under it:
(function Apod() {
var api_key = 'NNKOjkoul8n1CH1NoUFo',
url = 'https://api.nasa.gov/planetary/apod' + "?api_key=" + api_key,
data;
var apodRequest = new XMLHttpRequest();
apodRequest.onreadystatechange = function() {
if (apodRequest.readyState === 4 && apodRequest.status === 200) {
var response = apodRequest.responseText;
var parsedAPOD = JSON.parse(response);
data += parsedAPOD;
for (i = 0; i < parsedAPOD.length; i++) {
data += parsedAPOD[i];
console.log("Parsing lines: <br>" + parsedAPOD[i]);
}
}
apodRequest.open("GET", url, true);
apodRequest.send(null);
}
}());
JSON page parsing:
{
"date": "2016-11-05",
"explanation": "Shot in Ultra HD, this stunning video can take you on a tour of the International Space Station. A fisheye lens with sharp focus and extreme depth of field provides an immersive visual experience of life in the orbital outpost. In the 18 minute fly-through, your point of view will float serenely while you watch our fair planet go by 400 kilometers below the seven-windowed Cupola, and explore the interior of the station's habitable nodes and modules from an astronaut's perspective. The modular International Space Station is Earth's largest artificial satellite, about the size of a football field in overall length and width. Its total pressurized volume is approximately equal to that of a Boeing 747 aircraft.",
"media_type": "video",
"service_version": "v1",
"title": "ISS Fisheye Fly-Through",
"url": "https://www.youtube.com/embed/DhmdyQdu96M?rel=0"
}
You have a few errors in your code.
First off, the general structure of it should be like this
(function Apod() {
var api_key = 'NNKOjkoul8n1CH1NoUFo',
url = 'https://api.nasa.gov/planetary/apod' + "?api_key=" + api_key,
data;
var apodRequest = new XMLHttpRequest();
apodRequest.onreadystatechange = function() {
//Code here
};
apodRequest.open("GET", url, true);
apodRequest.send(null);
}());
Notice how I moved apodRequest.open("GET", url, true); and apodRequest.send(null); outside of the onreadystatechange handler.
Secondly, instead of
apodRequest.onreadystatechange = function() {
if (apodRequest.readyState === 4 && apodRequest.status === 200) {
//Code here
}
}
you can simply do
apodRequest.onload = function() {
//Code here
};
As it's the event that will fire when a response is returned. So you don't need the if check inside.
Finally, inside the onload handler, you have a few mistakes, such as:
data += parsedAPOD; this is wrong because data is an object undefined up to this point and parsedAPOD is an object. Doing += between two objects will not merge them. If you want to merge two objects there are other ways, e.g. Object.assign
for (i = 0; i < parsedAPOD.length; i++) { ... } is wrong because parsedAPOD is an object. Objects do not have a length property, so this is not the correct way to iterate over it. Use for ... in ... loop instead
data += parsedAPOD[i]; is, again, wrong, because this is not the way to merge objects.
parsedAPOD[i] is wrong because i is an integer, in this case, so parsedAPOD[i] is simply undefined.
Hope this analysis helps you correct your code.
first of all parsedAPOD is an object and parsedAPOD.length is not valid. You can use for in loop to iterate through an object as below.
for (var i in parsedAPOD) {
data += parsedAPOD[i];
console.log("Parsing lines: <br>" + parsedAPOD[i]);
}
I have large number arrays in JS which I want to pass to C++ for processing.
IMHO the most efficient way is to let JS write directly to the C++ heap and pass a pointer as argument within a direct call, like:
var size = 4096,
BPE = Float64Array.BYTES_PER_ELEMENT,
buf = Module._malloc(size * BPE),
numbers = Module.HEAPF64.subarray(buf / BPE, buf / BPE + size),
i;
// Populate the array and process the numbers:
parseResult(result, numbers);
Module.myFunc(buf, size);
The C++ functions to process the numbers look like:
void origFunc(double *buf, unsigned int size) {
// process the data ...
}
void myFunc(uintptr_t bufAddr, unsigned int size) {
origFunc(reinterpret_cast<double*>(bufAddr), size);
}
That works as expected but I wonder if there is any chance to call the origFunc directly from Javascript to get rid of myFunc and the ugly reinterpret_cast.
When I tried to bind origFunc via:
EMSCRIPTEN_BINDINGS(test) {
function("origFunc", &origFunc, emscripten::allow_raw_pointers());
}
... and call it directly:
Module.origFunc(buf, size);
I get the error:
Uncaught UnboundTypeError: Cannot call origFunc due to unbound types: Pd
Is this a general restriction of emscripten or is there a "less dirty" solutions than the reinterpret_cast work around?
You can use a static_cast if you
specify that function takes a void * rather than a uintptr_t;
don't use EMSCRIPTEN_BINDINGS, but use the EMSCRIPTEN_KEEPALIVE + cwrap / ccall way of communicating JS->C++ . For some reason, the EMSCRIPTEN_BINDINGS way resulted in a getTypeName is not defined exception when I tried it.
So the function looks like:
extern "C" int EMSCRIPTEN_KEEPALIVE myFunc(void *bufAddr, unsigned int size) {
origFunc(static_cast<double *>(bufAddr), size);
return 0;
}
which can be called from Javascript by
Module.ccall('myFunc', 'number' ['number', 'number'], [buf, size]);
I have a website that uses AJAX to deliver a JSON formatted string to a HighCharts chart.
You can see this as the middle JSON code part at:
http://jsfiddle.net/1Loag7pv/
$('#container').highcharts(
//JSON Start
{
"plotOptions": {
"series": {"animation": {"duration": 500}}
,"pie": {
"allowPointSelect": true,
"cursor": "pointer",
"dataLabels": {"formatter":function(){return this.point.name+': '+this.percentage.toFixed(1) + '%';}}
}
},
"chart":{"renderTo":"divReportChart"}
,"title":{"text":"Sales Totals"}
,"xAxis":{"title":{"text":"Item"}, "categories":["Taxes","Discounts","NetSalesTotal"], "gridLineWidth":1}
,"yAxis":[{"title":{"text":"Amount"}, "gridLineWidth":1}]
,"series":[{"name":"Amount","type":"pie", "startAngle": -60,"yAxis": 0,"data":[["Taxes",17.8700],["Discounts",36.0000],["NetSalesTotal",377.9500]]}]
}
//JSON end
);
The problem is that the function part...
"dataLabels": {"formatter":function(){return this.point.name+': '+this.percentage.toFixed(1) + '%';}}
is not being transferred via the JSON
All research tells me that there is NO WAY to do this.
IE... Is it valid to define functions in JSON results?
Anybody got an idea on how to get around this limitation?
It is true that you cannot pass functions in JSON. Javascript is a superset of JSON.
A common approach is for the chart to be defined in javascript (e.g. during the page load), and the page then requests just the data via Ajax. When the data is returned it can be added to the chart object, either before it is rendered or afterwards using the highcharts API.
If you really want to pass the formatter function from the server with the chart, send it as a string, and then turn it into a function like this:
var fn = Function(mystring);
and use it in highcharts like:
chart.plotOptions.pie.dataLabels = {"formatter":fn};
I've re-factored your example to show the approach: http://jsfiddle.net/wo7zn0bw/
I had a similar conundrum. I wanted to create the JSON server side (ruby on rails) so I could create images of charts for a web API and also present it on the client web browser with the same code. This is similar to SteveP's answer.
To conform with JSON standards, I changed all formatter functions to strings
{"formatter": "function(){ return this.point.name+':'+this.percentage.toFixed(1) + '%';}"}
On the web side, I navigate the hash looking for formatter keys and replace them with the function using this code (may be a better way!?). javascript:
function HashNavigator(){
this.navigateAndReplace = function(hash, key){
if (!this.isObject(hash)){
//Nice if only navigated hashes and arrays
return;
}
var keys = Object.keys(hash);
for(var i = 0; i< keys.length; i++){
if (keys[i] == key){
//convert string to js function
hash[keys[i]] = this.parseFunction(hash[keys[i]]);
} else if (this.isObject(hash[keys[i]])){
//navigate hash tree
this.navigateAndReplace(hash[keys[i]], key);
} else {
//continue
}
}
};
this.isObject = function(testVar) {
return testVar !== null && typeof testVar === 'object'
}
//http://stackoverflow.com/questions/7650071/is-there-a-way-to-create-a-function-from-a-string-with-javascript
this.parseFunction = function(fstring){
var funcReg = /function *\(([^()]*)\)[ \n\t]*{(.*)}/gmi;
var match = funcReg.exec(fstring.replace(/\n/g, ' '));
if(match) {
return new Function(match[1].split(','), match[2]);
}
return null;
};
}
To use this, would be something similar to this javascript:
hashNavigator = new HashNavigator();
hashNavigator.navigateAndReplace(myHighchartsHash, "formatter")
At that point the hash/js-object is Highcharts ready
Similar idea was used for the web image API.
I was really hoping that hacking at the JSON was not the only solution, but it works!
I used a different approach. I created a JSON like below
{"formatter": "function(){ return this.point.name+':'+this.percentage.toFixed(1) + '%';}"}
When I came to evaluating the expression, I used (assuming that the value of the 'formatter' is formatterValueString)
formatterValueString = formatterValueString.replace('function()', '');
let opts = (new Function(formatterValueString)).call(this);
formatterValue = opts;
The reason to use this approach was it became hard to bind 'this' with the function. The eval() function did not go well with accessing variable this. I am sure there are ways to do it. Just thought this was quick.
I wonder what are the different and which is the best method to set up some multilingual javascript application. i want to have all used strings in one file to easily change strings or add more languages later.
thnx!
You can simply make a big object tree:
var languages = {
english:{
Save:"Save"
},
german:{
Save:"Speichern"
}
};
In your app:
var l = languages.german;
alert(l.Save); //Alerts "Speicher"
The benefit of this system is that you can make sub objects to group some values together.
Whatever you do, the most important thing is to separate between your code and the texts.
If the code and the texts are mixed, maintenance will be impossible and you'll soon abandon it.
The translatable texts must be easily scanned, so that translators can translate just texts. Then, you should be able to insert the translations conveniently.
We use a JS file that includes a map of strings. We have a simple Python script that extracts the translatable strings from it. The same script also builds the output JS file that includes the same labels with the translated strings.
The result is:
When the application evolves, it's
easy to add new strings.
The script automatically finds the new strings
and we can translate them.
Completed translations go back to the JS file
without manual work.
I like using a "language dictionary array", which you can do using JSON or a simple array.
This is easy to implement:
var lang = 0 //0 = english, 1=french
var phrases=[]
phrases['cancel'] = "cancel,annuler".split(",")
alert(phrases['cancel'][lang])
Here's a function I put together to handle language translations based on the accepted answer in this question:
/**
Core script to handle language translations
**/
var Language = function() {
var activeLanguage = 'en';
var languagePack = {
'en': {
'hello-world': 'Hello World',
'show-variants': 'Show Variants',
'hide-variants': 'Hide Variants'
},
'fr': {
'hello-world': 'Bonjour World',
'show-variants': 'représentent des variantes',
'hide-variants': 'masquer variantes'
}
}
var translate = function(key, language)
{
if (typeof languagePack[language] == 'undefined')
{
return;
}else {
return languagePack[language][key];
}
};
return {
init: function(language)
{
activeLanguage = language;
},
getString: function(key, defaultText)
{
var text = translate(key, activeLanguage);
if (typeof(text) === 'undefined' || text.length == 0 || text == null)
{
text = defaultText;
}
return text;
}
}
}();
Then to initialise it in common page code, where ${language.language} is jsp code to set the language from a server side configuration.
<script type="text/javascript">
jQuery(document).ready(function() {
Language.init('${language.language}');
});
</script>
Then whenever you need a message use
Language.getString('hello-world', 'Hi World');