How to download the complete project structure from cdnjs using Python - javascript

I want to download the complete project from the cdnjs cloud to local folder.
I have tried this:
import requests
files = requests.get("https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_HTMLorMML%2CSafe.js&ver=4.1")
with open("mathjax.js","w") as file:
file.write(files.text)
Now this download the js file. When I tried using the same code to get the project instead of the js file, the output was weird.
So I tried using the cdnjs and check what happens when I use cdnjs cloud and when I use local file.
I have got this difference as shown in the images:
Using cdnjs:
Using Local file:
How I can get the similar structure as I get when I use cdnjs?
Kindly, advise me.

The URL you are providing to requests module is just the URL of one file MathJax.js, that is why you are getting only that file as output.
What you want is to download the complete directory mathjax/2.7.5/. However, if we request the whole directory, the server forbids such requests.
An alternate approach is to get relative paths of all the files from the main directory, which you already have as you showed in image. You can then download each of the file independently and store it into its respective folder. You'll have the whole directory ready at the end.
Try the following code for this purpose.
import requests
import os
baseUrl="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/" #Base URL for the main directory
#List containing relative paths of all required files
relativePaths=['config/Safe.js?V=2.7.5',
'config/TeX-AMS-MML_HTMLorMML.js?V=2.7.5',
'extensions/Safe.js?V=2.7.5',
'jax/output/SVG/fonts/TeX/fontdata.js?V=2.7.5',
'jax/output/SVG/jax.js?V=2.7.5',
'MathJax.js?config=TeX-AMS-MML_HTMLorMML%2CSafe.js&ver=4.1']
parentDir='\\'.join(baseUrl.split('/')[-3:]) #Parent directory from URL
for path in relativePaths: #For all files
req=requests.get(baseUrl+path) #forming url
filename=path.split("/")[-1].split("?")[0] #extracting filename out of url
directory=os.path.join(parentDir,"\\".join(path.split('/')[:-1])) #Extracting directories path for local path formation
if not os.path.exists(directory): #Creating local direcories if they do not exist
os.makedirs(directory)
with open(os.path.join(directory,filename),"wb+") as file: #Storing results into files
file.write(req.content)
Local Directory Structure Output:

Beyond iterating over a defined list of files, you could also look at a couple of other options that could take a more dynamic approach to fetch files from the CDN.
cdnjs is powered by a GitHub repository, so you could explore cloning it and extracting files (I'd recommend use sparse-checkout if you do this due to repo size) or you could look at using the GitHub API to navigate the repository an extract files: github.com/cdnjs/cdnjs/tree/master/ajax/libs/mathjax/2.7.5
We actually have an API available for cdnjs, which allows you to rather easily get all the files within a version of a library. Using that list, you could then perform a similar iterative solution to what Hamza suggested to get a copy of all the files locally: https://api.cdnjs.com/libraries/mathjax?fields=assets (annoyingly we've not yet implemented API navigation per version)
Hope that helps!
Matt, cdnjs maintainer.

Related

What is the use for the special "public" folder in a Meteor app?

I'm currently using Meteor and trying to learn more about the framework. In the documentation about special directories the following is said about the public/ special directory:
All files inside a top-level directory called public/ are served as-is to the client. When referencing these assets, do not include public/ in the URL, write the URL as if they were all in the top level. For example, reference public/bg.png as <img src='/bg.png' />. This is the best place for favicon.ico, robots.txt, and similar files.
My question is: since I refer to files inside of public/ directory as if they were located in the root folder of my application, what's the different between putting the files in the public/ folder and in the root folder? Or is it just for organization sake?
Also the documentation I quoted above makes some examples using assets (some pngs and favicon.ico) and no JavaScript or HTML files. Am I able to put some JavaScript code in there and then import them in another file by referencing as if this code was located in the root of my app? Or is the public/ directory somewhat made only for assets?
I failed to find any docs that explains what is done to files inside this directory in detail (I only found what I quoted above). So if any documentation of that kind is available it would help a lot!
My question is: since I refer to files inside of public/ directory as if they were located in the root folder of my application, what's the different between putting the files in the public/ folder and in the root folder? Or is it just for organization sake?
Just because you can reference or "import" a file from public/ doesn't mean it functions in the same way to how a normal file import would work. Files located in public gets served as is without being minified/run through the Meteor pipleline. Second, these files are accessible to the client which makes sense given how'd import them without preceding slashes and keep them mostly to serve stuff like favicon and what not.
So in a sense, such files within public are made available within relation to your client bundle/code whilst not being a part of them, get it?
This way of serving assets isn't unique to Meteor, even React has a public directory.
Also the documentation I quoted above makes some examples using assets (some pngs and favicon.ico) and no JavaScript or HTML files. Am I able to put some JavaScript code in there and then import them in another file by referencing as if this code was located in the root of my app? Or is the public/ directory somewhat made only for assets?
AFAIK, you can have files of any type in public but since
It's served as is to the client, meaning it's exposed to the public
It doesn't get minified (i.e being part of the final application build code)
You're advised to not have any of the application code within this directory.
The Public folder is how you serve your static files, when you put a file in your root folder it will not be sent to the client by default and you can't use it in your css, when you put that file (say an image) in your public folder you can use it from the css and refer to it as if it was in your root folder, so if I put a.jpg in the public folder I can use url(/a.jpg) in my css, that won't work if a.jpg is simply in your root folder, that's what the docs mean when they say it's served as if it was the root folder.
unlike in Rails, Meteor initiatives don’t have a rigid document structure and you are quite a whole lot free to prepare your projects as you want. a few folder names but have unique which means, and documents within them will be dealt with in a different way.
consumer
files here will be loaded at the client simplest. files in that folder don’t need things like Meteor.isClient.
server
Loaded on the server best, duh! No need for Meteor.isServer whilst files are in that folder, the client won’t see these files.
public
This directory is for property like photographs. on your initiatives, you reference stuff in the public folder as if they have been in the root folder. as an example, when you have a report: public/nude.jpg, then for your app you include it with .
personal
files only available at the server facet thru the assets API.
checks
documents in there received’t be loaded anywhere and are used for checking out your app.
lib
documents in that folder are loaded earlier than whatever else, which makes it the best listing to vicinity the distinct libraries used on a undertaking.

Cannot create Vue application http-server - Error: css and js files 404 Not found

I want to dockerize my vue app, but when I run it in a docker container nothing is loaded in the browser.
Since I run CMD["http-server", "dist"] in my Dockerfile, I decided to test it out locally to troubleshoot the issue.
Running:
npm run serve
Works fine and I get:
Then I run
npm run build
I believe this is due to having a posters folder with 50,000+ jpeg images in the assets directory which I dynamically display in the app as follows:
<div v-for="movie in this.recommendations" :key="movie" class="movie-card col-md-2">
<img v-if="movie['poster']=='True'" :src="getImgUrl(movie['movieId'])" v-bind:alt="pic">
And the getImgUrl function is:
getImgUrl(imgName) {
var images = require.context('../assets/posters', false, /\.jpg$/)
return images('./' + imgName + ".jpg")
}
vue/cli suggests
webpack performance recommendations:
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/
but I'm not sure how to implement either one of these or
if hosting the images on a public google drive and importing them from there would solve the issue?
Pretty new to vue so any help would be much appreciated!
By using the assets folder and using require you are bundling all of your images into your code, by encoding them as base64. So when it is compiled it is creating GIANT chunks. Because the images are compiled into the source.
What you should do is move your images from assets to the public directory. Then load the images via HTTP. This means that your code and images remain separate. When the page loads, the browser requests the images separately from your code, and loads them into the file.
For example
<img v-if="movie['poster']=='True'" :src="getImgUrl(movie['movieId'])" v-bind:alt="pic">
getImgUrl(imgName) {
return `/posters/${imgName}.jpg`
}
Thus your directory structure would become
-public
-|--posters
-|--|--Poster1.jpg
-|--|--Poster2.jpg
-|--|--Poster3.jpg
-|--|--Poster4.jpg
etc
The public directory overly simplified acts as a webserver. Anything that is in it can be accessed directly. For example, if you were to move your images over, to the public directory, with the directory structure above, and access localhost:8080/posters/Poster1.jpg, it would load just the image, without even needing a Vue router.
For a better, in-depth description of the public folder, and the role it serves check out the docs.

Using webpack and publicPath for static resources

The following question was rewritten, because I have now a working solution, but no answer to the question above.
The repository that shows different scenarios how to use resources packed with webpack is named example-webpack-dynamic-resources. It contains 3 modules:
inline: a solution, but not useful in my context (many resource files)
file: a solution by using the plugin webpack-require-from
public-path: no solution yet, shows how I would like to use __webpack?public_path__.
I think I have read any resource about webpack and publicPath and __webpack_public_path__, but I don't get it to work. I try to dynamically change the path to static resources, but it fails.
Here is my context:
I build a Javascript library that will be used on web pages (HTML, CSS, Javascript).
It provides a lot (>100) static resources to small image files, combined > 500 KB. Only a fraction of it will be used by the user looking at the web site.
Therefore I would like to pack the CSS into the bundle, but keep the image resources in a directory located on the server somewhere. The default path to it will be /img.
As long as I use the same structure (which means, images only under ROOT/img/**, everything is ok.
But the users of the library should be able to configure the path to the image resources on their will.
You will find all relevant files in my example repository example-webpack-dynamic-resources in the module public-path-resources.
webpack.js: Use file-loader for images, which are referenced in CSS files. CSS will be inlined by style-loader and css-loader.
src/public-path.js: Define the global variable with a default (no environment variable).
src/index.js: require first public-path, then the logic.
examples/exam1-root/index.html: Tries to use the assets in the sub directory lib, sets the value therefore to __webpack_public_path__ = '/lib/. Not working.
examples/exam2-different-dirs/index.html: Moves the library to a different dir (not relevant), but uses the originally defined directory pgnv-assets for the assets. Working.
examples/exam3-non-standard-dirs/index.html: Try to use instead my-assets as directory for the assets. Not working.
How could the __webpack_public_path__ defined at runtime in the index.html file?

serving static content in code structure generated by yeoman for angular fullstack

I am following the code structure generated by yeoman for angular fullstack.
I want to include a script called core.js in file called app.html.
<script src="core.js"></script>
I do not see express.static anywhere in this for serving static files.
I tried using it but it did not help.
It can not locate it and gives 404.
How do I get around this ?
It had happened before as well but I could get around it by using express.static and serving files from location pointed by it.
It did not help this time though.
Update:
I have app.html in folder called Music. In same folder, I have a sub folder called js where I have placed my core.js file that is to be included in app.html. I tried to access it using absolute as well as relative path but did not help and still gives 404.
In angular, the scripts go in the relevant subfolder of /scripts. Either in /controllers, /services/, /directives, etc. You then reference them in your html as such:
<script src="scripts/controllers/core.js"></script>
As for express.static, express is a NodeJS wrapper for HTTP. So that will be the service you create that lives on some Node server remotely. express.static allows the Node server to deliver static content files from the file set at the remote server. It does not go in your angular application.

How do I make text files accessible to the server on Meteor

I'm surprised I can't google my answer here... it seems no one else is having the issue.
When you run the meteor service the js, html, etc. is packaged in the .meteor/local/build folder, but it appears to exclude stuff that isn't js or html. I have a folder called "magicsets" and one called "magicimgs" and neither are in the /local/build folder. This is obviously why, when i attempt to use fs to readfile, it fails to find the file "magicsets/M14.json"
I tried putting the magicsets folder into a folder named "private", but that didn't accomplish anything.
How do I make files accessible locally to my server via FS and how do I make files accessible publically to my server via raw urls?
I'm sure I'm missing something very simple, because there are lots of more complicated questions and answers on SO, yet there is no answer for this. Thanks.
Meteor 0.6.5 which was released yesterday has a new feature which helps loads with this.
Make a directory called /private which you can access with the new Assets.getText or Assets.getBinary functions.
The stuff in the /private directory will then be bundled up into a directory called assets in /program/server/assets and it will not be accessible to the web & you wouldn't need to worry about using fs either. You could just use Assets.getText instead
To make a publicly accessible file put it in /public. So if you had a.jpg at /public/a.jpg it would be accessible at http://yourdomain.com/a.jpg
If you want text files to be available to the webserver i.e. the server that defaults to port 3000, create a folder called public in the root of the project/app directory. drop your folder and files there. You would then be able to access them as http://localhost:3000/magicsets/M14.json
update: it looks like can override the bundler, but it does require changing some of the core code there's no .meteorignore file yet. check this SO answer out: https://stackoverflow.com/a/16742853/105282
To serve a directory of files publicly independent of what Meteor is doing, you can use the following approach. I do this, for example, when I need to link an entire (Javascript) git repo into my Meteor app so I can work on a checked out version of the library.
The following works for 0.6.5. It basically servers up a checked out folder of OpenLayers in /lib:
connect = Npm.require('connect')
RoutePolicy.declare('/lib', 'network')
WebApp.connectHandlers
.use(connect.bodyParser())
.use('/lib', connect.static("/home/mao/projects/openlayers/lib"))
For more information, see https://github.com/meteor/meteor/issues/1229.

Categories

Resources