The Goal:
Optimize a web page that needs to display large amount of text data (25mb+) in a textarea or content editable div, without losing too much performance.
Currently, loading a 10mb file takes ~3 seconds on Chrome. I would like this to be 1 second max.
Ideas and Attempts:
I am loading local text files from a users computer using <input type="file"> and have no problem loading large files directly into memory. However as soon as I try and display this text data in a textarea, I naturally run into performance issues.
I have spellcheck, auto capitalization, and auto complete all disabled, and this certainly helped, however I would like to minimize the amount of lag when attempting to render large files (files greater than 10mb, max of 100mb would be ideal).
<textarea autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false">
One idea I had was to only render say 100 lines before and 100 lines after the current line, and when the users scrolls the textarea, I would then just switch out what data is being displayed. I can swap a few hundred lines without any noticeable lag, but hundreds of thousands locks up the entire page.
I was also looking at projects such as CodeMirror which is used in some javascript based text editors, and chrome dev tools. However a quick test showed similar performance issues when initially loading large amounts of text.
Another idea was to use highlight.js to render the text dom elements, but I was also noticing large amounts of large when dealing with thousands of DOM elemetns.
This site seems to tackle a similar example by dynamically creating and displaying the dom elements, instead of attempting to render all at once.
I have found that if the number of records in the grid becomes more then just a few thousands the grid gets very slow because the rendering speed is directly related to the number of nodes in the DOM. If the number of nodes in the DOM is more then 40-50k (depending on your computer configuration and amount of memory), your browser will crash or will become unresponsive.
So, I decided to set out on a quest to do two things: (1) dynamically create records as user scrolls (2) optimize grid to handle large data sets. After a few weeks of work, the grid was optimized and ready for testing.
I think this is similar to my first idea, but I have not tried this approach yet.
I'm hoping someone who's had experience with something similar might be able to offer some advice on which path to take, or offer some additional ideas.
Before anyone asks, I cannot not show this data, it needs to be user editable, it does not need to highlight code, or show line numbers. Lastly, the entire text file is being loaded with a FileReader into a variable.
I would like to avoid uploading the file to my webserver if possible for end user privacy and NDA concerns.
Server config: Ubuntu 16.04 LAPP Stack with Laravel 5.4, open to NodeJS solutions though. Use of jQuery is allowed.
Proposed Solution(s):
Lazy Loading - only displaying say 300 line "chunks" at a time as the user scrolls. Would need to make the scrollbar the appropriate height ahead of time in this case though. - Also, should unload these "chunks" as the users scrolls to reduce total the DOM rendering load.
Pseduo code:
c_chunk = scrollpos / scrollheight * totalChunks;
chunk_block = chunk[c_chunk--] + chunk[c_chunk] + chunk[c_chunk++];
my_textarea.val(chunk_block);
Any thoughts on this method?
Thanks all.
As promised here's my solution:
So after fumbling around with various ideas and trying different methods, I think I finally settled on using Ace for my project. I chose Ace for a few reasons.
Well documented API that was easy to understand in addition to a fairly large and active community.
Performance, Ace hands down gets far superior performance compared to my test with Code Mirror using large files. (500,000 lines, github says Ace has been tested with up to 4 million lines.
Features, out of the box Ace has more features to get you up and running with little user setup or configuration. Getting started took less than 10 lines total!
It doesn't support as many browsers as Code Mirror, but my project is running with WebGL so I was not very concerned.
So you understand my reasoning, and you want to get Ace up and running yourself? Easy, just hop on over to the build repo and grab one whichever flavor suites your needs. I grabbed the src-min release as I will just add it to my build script.
Then just include the javascript file in whatever way you do:
<script src="/ace/ace.js" type="text/javascript" charset="utf-8"></script>
And an element to your page with id="editor". In my case I'll attach it directly to a div:
<div id="editor">Any text you want to have auto displayed</div>
and in a javascript file:
var editor = ace.edit("editor");
editor.setTheme("ace/theme/chrome");
editor.session.setMode("ace/mode/javascript");
editor.$blockScrolling = Infinity;
If you just want to look around to see what languages/themes/options/etc. are available, just head over to Ace's kitchen sink page and play around with the editor.
Now I also needed a few functions, such as being able to load a file into the text area - and you can do that with the following:
Load local file:
// Add event listener to the file input
// Note: This will not fire twice if the user opens File A, then re-opens file A
// To detect re-opening a file you will need to clear the input
// after the file is read so that it "changes" upon re-opening
document.getElementById('open-file').addEventListener('change', handleFileOpen, false);
// First we create the function handler that fires when our input is changed
function handleFileOpen(e) {
var file = e.target.files[0]; // Get first file selected
// Load file using file reader function
loadFile(file);
}
function loadFile(file) {
// Create file reader and relevant Ace code
var reader = new FileReader();
reader.onload = function(e) {
// Get text contents of file
var data = e.target.result;
// Update Ace Editor with loaded data
editor.setValue(data, -1);
};
// Now that we defined the function we want to run, read the file
reader.readAsText(file);
}
The -1 in the setValue() function means place cursor at the beginning of the file (top) - 1 would be the bottom.
There are tons of events and properties you can hook into and mess in order to tweak this editor to your exact liking. They are all extremely straight forward and easy to use so if you're undecided, it's worth the time to try it out.
I wasted 2 days trying to get clusterize + highlightjs to work with my setup and gave up finally and replaced that entire setup with Ace in less than a day. Really impressed with editor so far!
Here's a link to their "How-to Guide": https://ace.c9.io/#nav=howto
and their API reference (which is extremely useful): https://ace.c9.io/#nav=api
Related
I have an html5 canvas project that's made to function as a multi-page eLearning interaction. Each "page" is designed to be a movieclip set on a main timeline, and buttons (page indicators) just use gotoAndPlay(); to get to the right page.
I ran into an issue after making it 5 pages long where Animate CC would say it was "Out of Memory" and couldn't export anything because the file was too big.
It seems like this question is on the right track, but I was trying to see if there'd be a way to divide each page into a separate .FLA file, and then combine them after?
I'd provide a link, but I can't publish it...
There's an intro page, then 3 scenarios the user has to go through. These should auto advance, but should be accessible from the other pages.
To further complicate it, the last page (conclusion/review) is supposed to link back to each of the three scenarios, shown here:
At this point, I'm not even sure if it's possible to get it to work how I want. It would have been nice if Animate CC could've just published it out, and I wouldn't have a problem.
As long as there are no duplicate names, you should be able to just load all the libs at once. The export format is built so it is additive -- so it will use existing lib, images, and ss references to combine all the lib elements into one global library.
You should be able to confirm this, and then combine/minify all the files together if it works.
I'm currently implementing a system for running build process remotely from browser. One of the tasks is to display build console output which downloads partially by ajax. And here I've faced the problem: I'm trying to output this log to textarea control and the log is quite heavy (may reach 12Mb and more depending on how many targets are building). My concern is about a compatibility of textarea with my requirements. I'm beginning to suspect this is not the best way of displaying so huge amount of info since I've been trying different approaches for appending data to textarea:
<textarea id='log'></textarea>
//using jQuery
$("#log").append(data["log"]); <- data["log"] is ajax result
//or classic JS
document.getElementById("log").value += data["log"]
the result is always the same: as larger textarea content as longer UI lags (it may entirely freeze for a second or two). Any ideas would be appreciated.
http://jsfiddle.net/93bexk59/
Hey,
basically - don't use textarea and minimize amount of dom manipulations.
What you can do?
1 Use documentFragment API to build parts of DOM out of it
2 Cache data which passes from ajax and drop it on page every N-th items
3 Render the things using async calls, not immediately.
setTimeout(function(){
//some render here
}, 0)
I've attached an example (jsfiddle).
Hope I understand you correctly, and with this approach (list instead of textarea you have more customization options).
I'm trying to load a "webpage" using the Android WebView but the images won't load. Firstly, I retrieve a text String that's formatted in Markdown. Afterwards, I convert the Markdown into HTML using Markdown4J, and then using the Android WebView's loadDataWithBaseUrl() function, I load the WebView with the html.
String html = null;
try {
html = new Markdown4jProcessor().process(mText);
} catch (IOException e) {
e.printStackTrace();
}
String css = "<link rel=\"stylesheet\" type=\"text/css\" href=\"blogstyle.css\" />";
final String htmlData = css + html;
body.loadDataWithBaseURL("file:///android_asset/", htmlData, "text/html", "UTF-8", null);
Everything in the webview displays and is formatted properly except the images. The images just won't show, and there are definitely image tags inside of the HTML, such as
<img src="https://www.random.org/analysis/randbitmap-rdo.png">
And I have no idea why my WebView wouldn't be able to display this image?
EDIT: The internet permissions in the manifest is already requested. And here is an example of a sample markdown string. I then convert this line into HTML.
"One of the huge problems today with web development is that it isn’t modular; because of how CSS and HTML work, it’s extremely difficult to just drag and drop an element from one site into another without some very strange side-effects. Shape and size can be different, resources can fail to load based on relative paths, and you have to import at least a .html, .css and .js file for each element. Even with all that, there’s no guarantee you won’t have CSS or JavaScript collisions.\n\n \n\nThe solution to this problem? [**Web Components**](http://webcomponents.org/). Now you should know up front that Web Components don’t work in IE and Safari, but they *are* under development.\n\n \n\nSo, what are web components? Well, essentially, they are plug-n-play modules for the web. The picture here is actually a great example. If you’re familiar with web design, imagine how long this would take to make, even statically. You’d have to position the images, set the roundness, give a height to the three main elements, set the font colors… Basically, it’d take you more than five minutes to get this up-and-running.\n\n\n\nSo what if I told you *all the code for what you see is two lines?* Just these two:\n\n \n\n\n <linkref=\"import\"href=\"gitcard.html\">\n <github-carduser=\"nasawa\">\n\n\nAnd that’s all. Just import the html and use the custom element! Using a new templating system called *Shadow DOM*, the browser holds a template called *github-card* with encapsulated styles and JavaScript. This way, even if you have elements named the same, there will never be any collision. And of course, you can have the *github-card* tag wherever you want, as many times as you want, and data will not transfer over.\n\n \n\nOn the other end, if you’re creating a web component, you can easily add as many customizable fields as you want to allow the user to modify your component at specific spots. For example, this is what the simplified Google Maps component looks like:\n\n \n\n\n`<google-mapslatitude=\"30.686603\"longitude=\"-88.046224\"></google-maps>`\n \n\nAnd a complicated Google Maps component looks like this:\n\n\n <google-maplatitude=\"37.77493\"longitude=\"-122.41942\"fitToMarkers>\n <google-map-markerlatitude=\"37.779\"longitude=\"-122.3892\"\n draggable=\"true\"title=\"Go Giants!\"></google-map-marker>\n <google-map-markerlatitude=\"37.777\"longitude=\"-122.38911\"></google-map-marker>\n </google-map>\n\n\n \n\nAs you can see, it is truly a great time to be in web development. Complex procedures are getting better and easier every day as we push closer to a true implementation of HTML5 and a unified multiplatform experience, and web components are a spearhead on that advancement. Look out for the next big thing – it’ll be here before you know it!\n\n \n\nChristopher Johnson \n Jr. Associate Programmer"
which would give me the following in HTML:
<p>"One of the huge problems today with web development is that it isn’t modular; because of how CSS and HTML work, it’s extremely difficult to just drag and drop an element from one site into another without some very strange side-effects. Shape and size can be different, resources can fail to load based on relative paths, and you have to import at least a .html, .css and .js file for each element. Even with all that, there’s no guarantee you won’t have CSS or JavaScript collisions.</p>
<p> </p>
<p>The solution to this problem? <strong>Web Components</strong>. Now you should know up front that Web Components don’t work in IE and Safari, but they <em>are</em> under development.</p>
<p> </p>
<p>So, what are web components? Well, essentially, they are plug-n-play modules for the web. The picture here is actually a great example. If you’re familiar with web design, imagine how long this would take to make, even statically. You’d have to position the images, set the roundness, give a height to the three main elements, set the font colors… Basically, it’d take you more than five minutes to get this up-and-running.</p>
<p><img src="https://www.ruralsourcing.com/wp-content/uploads/2014/09/blog-image-265x300.png" alt="blog-image" title="" /></p>
<p>So what if I told you <em>all the code for what you see is two lines?</em> Just these two:</p>
<p> </p>
<pre><code><linkref=\"import\"href=\"gitcard.html\">
<github-carduser=\"nasawa\">
</code></pre>
<p>And that’s all. Just import the html and use the custom element! Using a new templating system called <em>Shadow DOM</em>, the browser holds a template called <em>github-card</em> with encapsulated styles and JavaScript. This way, even if you have elements named the same, there will never be any collision. And of course, you can have the <em>github-card</em> tag wherever you want, as many times as you want, and data will not transfer over.</p>
<p> </p>
<p>On the other end, if you’re creating a web component, you can easily add as many customizable fields as you want to allow the user to modify your component at specific spots. For example, this is what the simplified Google Maps component looks like:</p>
<p> </p>
<p><code><google-mapslatitude=\"30.686603\"longitude=\"-88.046224\"></google-maps></code>
</p>
<p>And a complicated Google Maps component looks like this:</p>
<pre><code><google-maplatitude=\"37.77493\"longitude=\"-122.41942\"fitToMarkers>
<google-map-markerlatitude=\"37.779\"longitude=\"-122.3892\"
draggable=\"true\"title=\"Go Giants!\"></google-map-marker>
<google-map-markerlatitude=\"37.777\"longitude=\"-122.38911\"></google-map-marker>
</google-map>
</code></pre>
<p> </p>
<p>As you can see, it is truly a great time to be in web development. Complex procedures are getting better and easier every day as we push closer to a true implementation of HTML5 and a unified multiplatform experience, and web components are a spearhead on that advancement. Look out for the next big thing – it’ll be here before you know it!</p>
<p> </p>
<p>Christopher Johnson <br />
Jr. Associate Programmer"</p>
I am working on a simple Cordova app with about 4 page types and I am trying to think through which is the better way to handle the inner HTML templates.
Hidden HTML hard coded into the HTML files that is hidden and populated/revealed by my JS.
Using a JS template system and appending and removing from the DOM.
I feel that appending all that to the DOM for a page is inefficient when I could just update the sections that change. But perhaps an append is lightweight enough where I shouldn't worry about it.
There are a number of ways you can do it. In terms of load on the browser. That is hard to say. From your question it is hard to know what is in these pages, what are you displaying, is it live data, static html etc.
When you first plot out an app, if you are from the old class of building multiple page websites, it can be a little concerning as to how well your app/page will run with all those pages crammed in to one, and all that data plus code.
The answer is, amazingly well. If done properly in modern browsers, and for example Ipads the app will run to near native performance.
The options you have are
Map all the pages into one HTML document. Hide each page content using css display:none, flip them into view using css animation, fading or just display:block.
Use a javascript routing library to map urls to blocks of code that deal with each page, this makes mapping out your app much easier, and means that buttons can just link to your pages, like a real website. see http://projects.jga.me/routie/
Building all the page templates into one page can make it hard to code, as the page becomes enormous, consider breaking the inner content of each page into separate files, you can then give each page holder a url and use a small xhr request to load the page on-the fly, once loaded you can cache it into memory or even local-storage, depending on whether you remove it when it is closed or keep it hidden.
In my experience you can put an enormous number or nodes into one page and have very little speed drop, bear in mind if you use something like jquery and do a lot of $(".page > .page1 > .items li") your going to have a slow app.
Tips
Use element ID's everywhere document.getElementById(..) is 100's of times faster in a loop that $(...)
cache elements when you find them, if you need them later store them in a memory cache.
keep for loop inner code to a minimum.
use a decent click touch libary like http://hammerjs.github.io/ and delegate all the events of the body tag or at least on each page.
If you need to touch the server, load data, think dom first, device second server later. A good app is a responsive app, that responds to the user instantly.
I know this has been posted a while ago, but for the sake of the users I am going to add my answer.
I completely agree with MartinWebb but my answer will shed some light on the results of his options. I am currently working on a similar project. Please note that this answer pertains to cordova (previously called phonegap) specifically. My app has about 5 pages with +-20 different components (input's, div's, h1's, p's, etc.). This is what i tried and the result of each:
jQuery was my first option, mainly because it is easy to use and reduces the amount of code required to accomplish a said goal. Result: First time I tried this approach I though I would spice it up with animations and transformations. The result of this was a very unresponsive app. I removed the animation and transformation, however due to the nature of my application I required multiple dynamically added components and jQuery just wasn't up for the task.
Css display:none and visible:hidden was my next option. I used javascript's dom to display certain div's. Result: This works if your not planning on switching many div shortly after one another eg. a simple menu. It quickly became apparent that this wasn't going to work. Also this does not eliminate my need for the dom. Remember document.getElementById('menu').style.display = "none"; is still part of the dom. This as a solution, for me, is poor. There is a reason that var menu= document.createElement('div'); is part of the language. Which brings me to my last option.
Building a page 90% on javascript's dom was my last option. Logically I could not see how 600 lines of code cold trump one .innerHTML or .style.display = "block"; but it did. Result: It was by far the most responsive of all the solutions.
I'm not saying that all webpages should be coded with dom appending, but as I stated previously, for a cordova app of a few pages (<6), with a few components a javascript dom appending approach would be best. It takes longer to code, but you will be rewarded with control and efficiency. I would suggest coding the backbone of your app in html and populating and controlling with javascript's dom.
Best of luck.
The first option, <div>s with display:none; would be more efficient by a small margin, but you can get the best of both worlds by compiling your JavaScript and templates together into a single file using something like browserify or require.js.
Employees need to check data imported using OCR. They have web interface showing imported data on the left side and the scanned image on the right. When dealing with a large list of items they get lost and are asking for a tool/pen able to highlight (PDF) rows that are already checked.
Is there any chance to use JavaScript and draw to the PDF that is viewed?
If you are viewing the PDF in a plug-in, the answer is no, not through JS.
If, however, you can leverage a tool like pdf.js, this may be possible, depending on what your document uses in terms of PDF elements. For example, pdf.js has an Annotation plug-in, which looks like it may enable you to actually create annotations.
You will not be able to save the changes, however. If you want to do this, you are strongly encouraged to use a proper UI and not just a PDF (though, you may read from the PDF to generate it).