Node.js Automate Login and Upload - javascript

I currently have a node.js script that automatically creates a group of files and then zips them ready for being uploaded on a site. I'm trying to add one extra piece of functionality to the script that will log into the site and upload the file itself.
I've done some reading around and found a lot about headless browsers but not sure if that's the right path to go down as they seem to rely on other applications like chromium and they're focused on testing sites.
Does anyone know where I should start looking?

In my current project I am using the following library from Google, puppeteer. I personally found it to be very easy to use, and it even provides access to the dev protocol that Google Chrome has.
I've done some reading around and found a lot about headless browsers but not sure if that's the right path to go down as they seem to rely on other applications like chromium and they're focused on testing sites.
Yes, they are often used for testing, to assure that the correct things are rendered on screen etc. However, in many scenarios, like yours, the use of a headless browser to interact with a website is totally legit in a non-testing scenario.

Related

How do I use AVIF images in React Single page Web App

Final Update
For most practical purposes, this question is obsolete as both firefox and chrome have native support for avif through the standard picture html tag with a source marked as type="image/avif". See https://reachlightspeed.com/blog/using-the-new-high-performance-avif-image-format-on-the-web-today/ . Fire fox still likes to hang and often forces a control F5 to bypass caches and requires sending the correct content type from the server. Hopefully will be fixed soon.
Here is the commit where I got avif support working: https://github.com/quackack/quackack-comics/commit/f1a98ed1f40b6a22584d61bc338bd91df3232fa5#diff-e25b0950ce48f4e928f98e0a6fbb694c . Note that it contains many unrelated changes and in fact avif is barely mentioned, only as a content type and a file extension.
Original Question
I am trying to change a website where I host web comics from using jpegs to the newer avif image format. It is much smaller and seems to be the new image format with the most widespread support. Unfortunately, web browsers don't properly support the new format yet. So I was planning to use this package: https://github.com/Kagami/avif.js to allow my comics to be rendered. Some basic tests showed that AVIF would give the same quality as jpeg for less than half the space.
Unfortunately, after more than 5 hours of time spent on this, I am unable to get this to work with my react framework. You can see my website at the time of writing at 'https://github.com/quackack/quackack-comics/tree/cda4c3893d8477192c4ff3aa78d00096b7621ff7'.
I tried using npm install to install avif.js and then added
require("avif.js").register("/avif-sw.js");
to index.js . But I get error
Failed to register/update a ServiceWorker for scope ‘http://localhost:3000/’: Bad Content-Type of ‘text/html’ received for script ‘http://localhost:3000/avif-sw.js’. Must be a JavaScript MIME type
And avif files are still not able to load. I think the requests are getting rerouted to index.html instead of the javascript package. It seems like the appropriate thing to do is something like
import * as avif from 'avif.js';
avif.register('avif.js/avif-sw.js');
But this fails too with the same error, as do many other similar variations.
At this point I am inclined to wait for proper browser support for avif, as I don't get enough traffic to worry about data costs anyway. If this could easily be fixed, then I would love to have the improvements from avif. I just want smaller file sizes and widespread browser support.
Update
Okay, I found that I could get this to work if I changed from the default react bundler (which I believe to be webpack) to Parcel. Then it does work exactly as you expect... until I try to deploy the project.
There is an issue where I cannot load the service worker when i try to deploy my single page webapp to AWS as a single page web app. There it makes a request to my url with avif-sw.js where there is not actually a js file. I believe the issue is closely related to https://github.com/parcel-bundler/parcel/issues/670
So the first key is to use Parcel to build your web app. But Parcel still does something wrong with deployment it seems. I will continue to investigate this in a few days.
Here is the almost working version using parcel: https://github.com/quackack/quackack-comics/tree/parcel
Update 2
My earlier update was incorrect. I only thought it was working because of a cached service worker. My final solution is in the answer below.
It doesn't seem to be a problem now. Simply convert your images to avif with tools like https://avif.io/ and use them as the image source or background source via typical CSS. As Chrome and Firefox now support it (even though users still have to enable it on Firefox), everything goes well. Even works on mobile now! :)
Okay, I finally got it working. Unfortunately, mobile devices don't seem to be able to handle the large file sizes so I had to keep using jpeg anyway. It worked on my laptop though.
Here is the commit that got everything to work: https://github.com/quackack/quackack-comics/commit/75e75307e688f0e515b4bbc9eb22eef290d2c209
What I had to do:
Switch to Parcel.
Copy the contents of the avif.js library into source before building. I used the command:
copyfiles -f node_modules/avif.js/*.js .
Put this specifically into reg.js:
require("avif.js").register("./avif-sw.js");
navigator.serviceWorker.register("./avif-sw.js", undefined);
What the last two steps do is trick parcel into actually keeping a copy of "avif-sw.js" around that can actually be loaded as a service worker. Probably with a bit more tinkering you can get this to work without using parcel at all, just by copying local and then registering. No requires required. But I stopped investigating after I found this solution can't work on mobile.
This was exceptionally hard to debug because service workers are cached by the browser and I had to clear broswer data after every edit. It was also hard to debug because the source files are cached to so I had to delete my projects cache and build frequently too.
You might also want to use npm module "http-server-spa", or similar, to test how your built SPA will act when deployed.

Scan and access local file directory in Firefox and IE?

I'm doing some research on whether or not it's possible for a web app (meant to be used and distributed internally) to scan and read files from a local directory (on user machine). I came across a couple of terms as following:
NPAPI: no longer supported by majority of web browser
ActiveX: IE only
Sandbox: Chrome uses this kind of technology, plus it's not fitting to the requirement so I have to look elsewhere
I feel like ActiveX might be the only option even though I haven't actually written any ActiveX control before (not sure if it's possible).
Also the goal is to support more than one kind of web browser, so other than IE I thought Firefox might be capable of achieving the requirement, since no search result so far said otherwise.
Could someone please give me some pointer? I just need to know if it's at all possible to build a ActiveX control or Firefox extension to scan and read files from a local directory. If it is, then what is the downside other than security vulnerability.

Copy entire website to iPad

I've created a website for a client's salesforce to use in hospitals, where they can't use wifi to access the internet as it can interfere with sensitive medical equipment. The website uses HTML, CSS and Javascript and does not require any form of server to work, which is fine when run locally on a laptop. However, they now want to use iPads and I've been trawling the Net for a simple solution in which I could copy the entire site to an iPad, while maintaining the site's directory structure, and then launch it locally with the browser. Nothing that I've found seems to be able to do this, although it seems such a ridiculously straightforward thing.
The best I've been able to do is to email the zipped site to my iPod, and then use the FileManager app to unzip and run the site in its own browser. It's useable but not great (every time you tap something you get an overlay at the top of the page) and there's no way I can find of adding a shortcut to the desktop - is there a better way of doing this?
You can look into something like React Native which allows you to write javascript HTML applications and deploy them as native iOS and Android applications. It shouldn't require too much work to move it over enough to just compile it and run it as you do currently.
https://facebook.github.io/react-native/
Cordova to package it in to an app https://cordova.apache.org/

Remote Debug Website

Is there a way to remotely debug a website?
I've just finished putting together a website that has some jquery animations. The site works fine on every machine/configuration I've tested it on.
One of the people the site needs to work for, however, reports that the animations don't work; which effectively breaks the website.
I strongly suspect his companies' network is the root of the problem; however diagnosing this is challenging as he is not a technical user and guiding him through the webkit inspector/console, etc. is not really an option.
Ideally I'd like to be able to 'capture' the network/javascript logs from IE or Chrome so that I can inspect them and attempt to work out what's gone wrong.
Aside:
I'm using an off-the-shelf Wordpress theme (http://theme.co/x/) for the site; so I expect the code is good.
While it doesn't seem possible to remotely capture and inspect the network or javascript logs from another machine's browser; there are a number of services that allow you to add automatic error reporting to your javascript code, which you can then inspect to find the root of the problem.
Examples of these are Errorception and Raygun.
As far as I have found, there aren't any similar tools to do so for monitoring network performance / loading specifically- although a similar approach with a custom script to detect if specific items have been loaded could be written.

Best ways to develop painlessly in Javascript on a local machine

I'm pretty new to workign with Javascript.
In most languages you can run the code quickly locally on your machine. From what I've seen, in JS you generally only use it via the browser, and so I've been uploading my code an viewing its effects in the browser. This has proven very tiresome. Also, if I mak one error, it seems like my JS/JQuery will just do NOTHING, instead of giving me a useful error, message, which is making it painfully slow to code in.
IS there some way to run JS locally to see that it is working as I go? And then only upload it to the web when I'm mostly done? What ways are there for me to do this? What ways aer there for me to unit test the Javascript locally? Say I have some JAML that should render as <p>HI</p>, how do I run this locally in a unit test?
Thanks for the help,
Alex
EDIT:
Thanks for all the great suggestions. I'll have to take a bit of time and go through them to see which ones best help me in my situation.
Since you're using jQuery, I assume that you actually want to manipulate the various elements on your page. So depending on your specific development enviroment, uploading it each time is probably the way to go anyway. If you can set up a dev enviroment on your local machine (not always possible) then go with that.
As an actual answer to your question, I suggest using Chrome's developer tools, it doesn't just have the console, but an element inspector, and a resource tracker (resource tracker is invaluable when working with JSON and AJAX, since invalid json will fail silently)
As far as I know, the firebug plugin for firefox (dont use it myself) has a similar feature set, so if you're more comfortable with that go with it.
Just remember, as a developer, your development (and debuggin) enviroment is just as important as the code that you are writing.
EDIT: Noticed that you mentioned unit testing. There are several unit testing frameworks out there for JS including one that integrates with firebug called FireUnit. Do a quick google search to find more if you want.
You don't need to upload the JS file to a server to test it. Just write an html and declare the js binding
<script
src="js/yourJSFile.js"
type="text/javascript"></script>
Edit the JS file in your favorite editor and then refresh the page to test it.
For unit testing the best option is Selenium. It allows you to record an interaction with the browser and then play it back.
You can use Firebug with Firefox to debug JS, and Google Chrome has a debugger built-in (use the Tools -> Developer Tools menu).
You can run Javascript from the local file on your machine in your browser, so you can skip the uploading step.
Also, I'd recommend using Firefox/Firebug combo for developing Javascript as it will be very handy, especially for the part you mentioned about not seeing what's going wrong with your code.
Even if you upload your javascript it gets downloaded back to you as soon as you visit the webpage that invoques it. Its run client side always. So stick to local and use firebug as the others have said. Google`s developer tool is quite nice too.
In the browser if you open the developer tools, follow the following steps:
1) Navigate to sources
2) Under sources, click snippet and open run.js
3) You can use run.js to write as much code as you want and run it locally only to see if your code is working or not (it will give you output on the console)
4) Also you can get used to some keyboard shortcuts to make it faster for you.
5) For small javascript codes, you can navigate to console and run your code there
If you want to do unit testing with Javascript there are extension of Firebug that can help you with that. I haven't try any of them, so I can't really tell you which one are worth considering, but you can easily find them if you search for the keyword "Firebug unit testing" on Google.
What seems to be comming on top is FireUnit. You can find some information about how it works here.
Consider Spider Monkey, which is a javascript engine separate from a browser. If what you are developing does not involve rendering to a webpage or can be separated from the rendering code (good practice!), then this could be useful.
I prefer Chrome to Firefox and I just found Web Server for Chrome.
It's just a Google App that quickly sets up a web server for you and will be set up anywhere you are logged into Chrome. It only allows file access to your current devices, or if you specify, other devices only on the current LAN.
You just point it to the directory with your index.html file and type http://127.0.0.1:8887 in your browser.
Additionally to the answers given you can use Jasmine for automated testing.
A tutorial that seems to help get started with automated testing on Jasmine is provided by Evan Hahn.
I used it and for me it works like a charm. Especially if test driven development is what you are going for!

Categories

Resources