JSDOM scraping on cloudfoundry doesn't work - javascript

I'm using the jsdom node.js module for an app that does some web scraping to get some data it needs. This works perfectly fine when I run locally.
When I push the application to cloudfoundry however, it crashes. The log is as follows:
====> /logs/stderr.log <====
/var/vcap/data/dea/apps/caretogethersandbox-0-8b20af9255bbf552d0f490cb60d0df55/app/node_modules/jsdom/lib/jsdom.js:171
features = JSON.parse(JSON.stringify(window.document.implementation._fea
^TypeError: Cannot read property 'implementation' of undefined
Is there something I'm missing here? I'm pretty stumped. I know the code I've written works fine, it just seems to be an issue between the module and cloudfoundry.

I was able to reproduce this problem pretty easily. It seems jsdom relies on a native compiled library called Contextify (https://github.com/brianmcd/contextify) and also has some bindings to Python from what I can gather. These are not supported by Cloud Foundry at present. There is a similar question on Github about this problem;
https://github.com/tmpvar/jsdom/issues/436
The last post at the time of writing this suggests the use of two 100% JS libraries called Domino and Zepto Node, they may well be worth checking out.

With only what you posted to go by its pretty much impossible to give you a direct answer.
But I have a couple of suggestions.
You have environmental variables that don't work the same in production as they do in development.
jsdom relies on jquery that is normally loaded-in externally.
Perhaps jquery is not loading properly.
Have you heard of cheerio. Its an alternative to jsdom that implements a subset of jquery and does not need to load jquery in externally. This makes much faster than jsdom.
You can watch this video created by the author for introduction to cheerio and learn more of its benefits.
Use the sample code on the cheerio github page and upload it your cloudfoundry account to see if everything works. Then you can make decision on weather to switch to cheerio.

Related

Angular 2+ server-side rendering with jQuery

I encountered something debilitating.
I'm pretty far in a project right now, and it definitely needs to be rendered server-side at some point.
The webapp lets users upload pictures such as profile picture and also "regular" pictures.
I'm using Croppie from Foliotek (https://github.com/Foliotek/Croppie) to do the cropping, which works well and is exactly what I need.
But Croppie relies on jQuery. Yesterday I found out that could be an issue if I want to make the app server-side rendered.
All the jQuery is used in ngAfterViewInit lifecycle hooks, so I'm hoping this makes it server-side rendering proof?
I know there is ng2-img-cropper available, but I'm not so keen to implement that one. BUT, if I can use ng2-img-cropper together with server-side rendering then I'll have to.
Or is there something far better I'm overlooking?
You can use jQuery on server, with Node, AFAIK. Just browse npm and search for suitable module. And do not forget, that Node have no DOM implementation, so you need module for virtual DOM also.
For example, you can use this jQuery wrapper with this DOM implementation. Example:
npm install jsdom
npm install jQuery
var jsdom = require('jsdom').jsdom
var myWindow = jsdom().createWindow()
var $ = require('jQuery')
var jq = require('jQuery').create()
var jQuery = require('jQuery').create(myWindow)
$("<h1>test passes</h1>").appendTo("body")
console.log($("body").html());
I found out there is an Angular Universal (server-side rendering) starter project available:
https://github.com/angular/universal-starter
I cloned it and implemented Croppie as I did in my development version and everything works just fine!
So... I wonder why I've read multiple times that things will totally break if using jQuery in your Angular 2+ SSR project. Also Max Schwarzmuller told so in his lectures on Udemy. Anyway. I'm happy to have found everything still works.
as i know, when ever jquery use "Window", ssr gets confused, because in server we don't have window(window is in browser).
but normally, jquery works.

GMLib Could not complete the operation due to error 80020101 (v1.5.3)

I have an application using the latest version (v1.5.3) of GMLIB. Starting today, the map now generates the above generic error when I place an object on the map. My application has not changed. I also tested the GMLIB MegaDemo that was provided and the same error is generated when starting the map - previously there was no error. I am using XE7 and have IE11 installed.
It appears the same error occurred several years ago and Catedill released an update to fix.
Below is the function and parameter I pass when the error occurs:
'MakeMarker'
'2,false,false,true,false,false,true,35.7519302368164,-121.285972595215,true,''001: SAMPLE'',true,'''',0,'''',false,0,0,0,true,''mtStyledMarker'',''siBubble'',''#0000FF'',''#FFFFFF'',''#00FF00'',false,false,true'
Anyone else just start getting this error?
Well, I have made a solution. I have found StyledMarker.js and I have uploaded it into the GitHub GMLib repository.
Now, you only need to do this changes:
Into .\resources\map.html file replace this url:
http://google-maps-utility-library-v3.googlecode.com/svn/trunk/styledmarker/src/StyledMarker.js
by this other:
https://cdn.rawgit.com/googlemaps/v3-utility-library/master/styledmarker/src/StyledMarker.js
or
https://cdn.rawgit.com/cadetill/gmlib_v1/master/Resources/StyledMarker.js
Recreate resource file executing .\resources\rc.cmd
Build (no compile) all GMLib
Another option is download changes from GitHub repository.
More info about the problem here.
Regards and sorry for the inconvenience.
Rather than using your own development copy of the Google utility script, I'd recommend changing the url in your .\resources\map.html from:
http://google-maps-utility-library-v3.googlecode.com/svn/trunk/styledmarker/src/StyledMarker.js
to the following cdn prefixed version of the source library:
https://cdn.rawgit.com/googlemaps/v3-utility-library/master/styledmarker/src/StyledMarker.js
In production, you really should be using the cdn version of the script as it has no traffic limits or throttling and the files are served via a super fast global CDN. However please bear in mind that, as a free service, it offers no uptime or support guarantees.
Accessing files hosted from Git is covered in more detail in the following SO answer:
Link and execute external JavaScript file hosted on GitHub
If you'd still prefer to use your own copy, for the same reasons I'd suggest using your cdn copy:
https://cdn.rawgit.com/cadetill/gmlib_v1/master/Resources/StyledMarker.js

NwJS, and requiring client libraries

I'm building a small app with nwjs, and naturally I want to use some typical client-side libraries. For example I have in my main app
var ko = require('knockout');
The problem is, while document is defined in my main app, it is not defined in the context of required libraries (I've checked). Interestingly knockout worked for quite a few things before I ran into an error where it was trying to access document.
This seems like a major problem for nwjs, unless I'm missing something. How are you supposed to use client-side libraries with nwjs?
(FWIW, there was an earlier question on almost the exact same topic, but it involved React.js which seems to have its own server/client behavior so the answers didn't address the basic issue.)
You have to use separate mechanisms to include libraries based on whether they are server-side or client-side. Server-side libraries can be loaded with require(). Client-side libraries (if they need access to the document environment) need to be loaded with <script> tags in the index.html file.
In hindsight it is obvious, but it took me the better part of a day to figure out, so posting in case anyone else has the same confusion...

Can you provide a module by another name?

In our application we use our own logging framework. Some of the external modules we use uses the debug module, which logs directly to the console.
I would like to wrap these logging calls, so that they are piped through our own framework.
I didn't want to override/hook console.log, because that's what our logging framework uses internally (and it feels dirty). So my next idea was to somehow instruct Node to load my own code when another module uses require('debug'). Is that possible? And, if so, how? Or is there maybe a better solution altogether?
For the time being, we just forked debug and we're now simply using our fork in our project.
We adjusted the fork to make use of our own logging solution. Integrating it was pretty easy, due to the support of GitHub URLs in package.json.

Javascript code

I checked some websites source code and JavaScript games. The problem is that everything is readable and understandable except for JavaScript code that is isolated on a file with the extension .js. It looks like this:
{Vargas=void0,h=!0,Ge=null,l=!1,AA=component,BA=Infinity,ca=set Timeout,DA=is Nan,m=Math,ea=deconstructionism;function He(a,b){return a.on-load=b}function IE(a,b){return a.on error=b}function ha(a,b) {return a.name=b}
As you can see, it's hard to read this code because of the stupid indentation. I tried to use Microsoft visual web developer and free JavaScript editors to organize the code, but it was all useless!
How can I make it more readable?
The best place to start is to look at other open source java script libraries/modules/plugins. You must have the original code though, because what you see in the browser is already "compiled" for the web to be small and fast.
For the client you have plenty frameworks. Look for example at the list jsfiddle uses (top-left). You can also use this tool to play with javascript without having to install anything. Search on the web for those projects (that jsfiddle uses as libraries) and look into the code.
There is also a server-side javascript library that allows you to write javascript code also for the server (also web apps, the server side of them). This is called Node.js Ceck this page to find out more. In Node.js you have almost an infinite number of open source and small modules: see the node module registry wehre you find the links also to the individual project.
In any case, you certainly need a Github account to play with other's code because most of these projects are stored on Github.

Categories

Resources