Meteor - Slow loading with large dataset - javascript

I'm working with quite large data in MongoDB and using it in my Meteor application. However, the size of the data is causing the webpage to load incredibly slowly.
The collection is around 17MB in size and contains 84,000 documents.
Using the Publish/Subscribe method I have the following code:
Imports -> Both -> MyCollection.js:
import { Mongo } from 'meteor/mongo';
export const statistics = new Mongo.Collection('statistics');
Server -> Main.js:
import { Meteor } from 'meteor/meteor';
import { HTTP } from 'meteor/http';
import { statistics } from '/imports/both/MyCollection';
Meteor.publish('statistics', function publishSomeData() {
return statistics.find();
});
Client -> Main.js:
import { Meteor } from 'meteor/meteor';
import { Template } from 'meteor/templating';
import { ReactiveVar } from 'meteor/reactive-var';
import { statistics } from '/imports/both/MyCollection';
import './main.html';
Template.application.onCreated(function applicationOnCreated() {
this.subscribe('statistics');
});
Template.application.onRendered(function applicationOnRendered() {
this.autorun(() => {
if (this.subscriptionsReady()) {
const statisticsData = statistics.findOne();
console.log(statisticsData);
}
});
});
So like I say this method works and the console logs the data. However, using an internet connection of around 60mbps it takes around 2 minutes to load the page and finally console log the data and sometimes I just get the 'Google is not responding' alert and I'm forced to force quit.
What is a more efficient means of loading the data into the application in order to avoid this terribly slow loading time? Any help would be greatly appreciated.
Many thanks,
G

Limit the amount of data you publish to the client.
Either only publish some fields of the statistics collection or 'lazy load' documents - pass a number of docs argument to the publication and use the limit option of find to only send that many docs to the client.
Alternatively, compile the data as needed on the server and only send the compiled data to the client.
Much more specific examples cannot be given without knowing the collection's nature.

Related

Initialise connections and batch service on app start

This query is specific to Next JS version 13 in Beta state.
I want to kick off a batch service to run every 24hours and also connect to mongo db
when the application starts.
In the previous versions, this could have been achieved by calling relevant functions within
the app.js file. If this was a node app, I could have called em inside the main index.js file.
How would we do this in NextJS 13? At present I am calling them inside the main pages.tsx file
which would lead to localhost:3000/. It does work.
But is this correct or is there a better correct way to do this?
// location /src/app/page.tsx
import { checkOrInitializeDbConnection } from '#/utils/database/connection';
import { kickOff24hourlyBatchService } from '#/utils/batch/kickOff24hourlyBatchService';
const Home = () => {
checkOrInitializeDbConnection();
kickOff24hourlyBatchService();
return (
<div>
Home
</div>
)
}
export default Home;

Best way to expose API on the front-end with ES6 modules?

I want to know the best way to expose my server's API to my front-end web app. I know I could place all the code in just one file and it'd work. But I want to know a more organized way to expose it. So I came up with two ideias:
First: Export a module named API and this module export its sub modules like Authentication module, users, tasks... The problem with this aproach is that I couldn't find a way to make dotted imports like import { API.authenticate }, I need to import all things from api and then access the object.
import { authenticate } from "./src/authenticationService.module.js";
import { users } from "./src/usersSerivice.module.js";
import { tasks } from "./src/tasksSerivice.module.js";
import { projects } from "./src/projectsSerivice.module.js";
import { projectSteps } from "./src/projectStepsSerivice.module.js";
export const API = { authenticate, users, tasks, projects, projectSteps };
authentication.module.js;
import { API } from "../backend/api.module.js";
API.authenticate.login("email", "pass");
Second: A module for each endpoint like AuthenticationAPI, UsersAPI, TasksAPI... but the problem is that I think I'm creating to much repetitive module names.
An example:
export const authenticateAPI = { login };
authentication.module.js
import { authenticateAPI } from "../backend/authentication-api.module.js";
authenticateAPI.login("email", "pass");
Is any of these a good approach or you suggest a better one? I'm new to programming.

How do I authenticate a Vue.js progressive web app with the Microsoft Graph

I have a Vue.js app. This app is a progressive web app, so it's intended to primarily run on the client-side. However, during the initial start-up, I need to authenticate the user in my Azure Active Directory, get data associated with their account, and store it for offline use.
I have a server-side API in place already for retrieving the data associated with a user account. I also know how to store it for offline use. However, my question is: how do I authenticate with the Microsoft Graph from my Vue.js app? Everything I see relies on using Node.js middleware, but unless I'm misunderstanding something, my progressive web app isn't a Node.js app. It's just straight up JavaScript, HTML, and CSS.
If the user closes the app, then revisits it in a couple of days, I believe I would need to use the refresh token to get a new access token. Still, once again, everything I see relies on Node.js middleware. I believe I need a solution that works purely in Vue.js / JavaScript. Am I mistaken?
Updates
1) Installed the Microsoft Graph Client via NPM (npm install #microsoft/microsoft-graph-client --save). This installed v1.7.0.
2) In my Vue.js app, I have:
import * as MicrosoftGraph from '#microsoft/microsoft-graph-client';
import * as Msal from 'msal';
let clientId = '<some guid>';
let scopes = ['user.read'];
let redirectUrl = 'http://localhost:1234/'; // This is registered in Azure AD.
let cb = (message, token, error, tokenType) => {
if (error) {
console.error(error);
} else {
console.log(token);
console.log(tokenType);
}
}
let reg = new Msal.UserAgentApplication(clientId, undefined, cb, { redirectUrl });
let authProvider = new MicrosoftGraph.MSALAuthenticationProvider(reg, scopes);
The last line generates an error that says: export 'MSALAuthenticationProvider' (imported as 'MicrosoftGraph') was not found in '#microsoft/microsoft-graph-client'
The last line generates an error that says: export 'MSALAuthenticationProvider' (imported as 'MicrosoftGraph') was not found in '#microsoft/microsoft-graph-client'
This error occurs because the main script (lib/src/index.js) of #microsoft/microsoft-graph-client does not export that symbol. You'll notice that logging MicrosoftGraph.MSALAuthenticationProvider yields undefined. Actually, the main script is intended to be run in Node middleware.
However, #microsoft/microsoft-graph-client provides browser scripts that do make MSALAuthenticationProvider available:
lib/graph-js-sdk-web.js
browserified bundle (not tree-shakable)
sets window.MicrosoftGraph, which contains MSALAuthenticationProvider
does not export any symbols itself
import '#microsoft/microsoft-graph-client/lib/graph-js-sdk-web'
let authProvider = new window.MicrosoftGraph.MSALAuthenticationProvider(/* ... */)
demo 1
lib/es/browser/index.js
ES Modules (tree-shakable)
exports MSALAuthenticationProvider
import { MSALAuthenticationProvider } from '#microsoft/microsoft-graph-client/lib/es/browser'
let authProvider = new MSALAuthenticationProvider(/* ... */)
demo 2
lib/src/browser/index.js
CommonJS module (not tree-shakable)
exports MSALAuthenticationProvider
import { MSALAuthenticationProvider } from '#microsoft/microsoft-graph-client/lib/src/browser'
let authProvider = new MSALAuthenticationProvider(/* ... */)
demo 3

meteor: import directory - modular import of methods which are needed

I'm migrating my meteor application to the import-function of meteor 1.3.
But I think this is not quite the best way it should be done. Isn't it possible to load/import just the method which is really needed?
I mean, right now just all methods are loaded by importing the the methods.js. But I would like to do that in a modular way. So if the form .fomNewElement is used in the app, the method insertArticle will be imported and so on. Not just loading everything...
Below you can see my folder structure for /imports and some content of the files. Is there anything more I could improve in the structure itself?
Also it would be great if the import would depend on user roles. Is this possible?
imports/api/article/client/article.js
import { Articles } from '../';
import { insertArticle, updateArticle } from '../methods.js';
Template.Articles.helpers({
// some helpers
});
Template.Artilces.onCreated(function() {
// some code
});
Template.Artilces.onRendered(function() {
// some code
});
Template.Articles.events({
'submit .formNewElement': function(event) {
event.preventDefault();
var title = event.target.title.value.trim();
insertArticle.call({
title: title
});
},
'click .anything': function() {}
});
As you can see, I put into that js-file all helpers, events and onCreated/onRendered code. Hope this is 'correct'... Please give me some hint, if this isn't very smart.
imports/api/article/index.js
export const Articles = new Mongo.Collection('articles');
imports/api/article/methods.js
import { Articles } from './';
export const insertArticle = new ValidatedMethod({
name: 'article.insert',
validate: new SimpleSchema({
title: { type: String }
}).validator(),
run( document ) {
Articles.insert( document );
}
});
export const updateArticle = new ValidatedMethod({
name: 'article.update',
validate: new SimpleSchema({
_id: { type: String },
'update.title': { type: String }
}).validator(),
run( { _id, update } ) {
Articles.update( _id, { $set: update } );
}
});
And the other files:
imports/startup/client/index.js
import '../../api/redactor-article/client';
imports/startup/server/index.js
import '../../api/redactor-article/server/publications.js';
import '../../api/redactor-article/methods.js';
imports/api/article/client/index.js
import './article.html';
import './article.sass';
import './article.js';
Filestructure
/imports
/api
/article
/client
article.html
article.js
article.sass
index.js
/server
publications.js
index.js
methods.js
Update
Maybe it would be a better way to structure an import module like this:
imports/
api/
articles/
publication.js
methods.js
collection.js
ui/
articles/
article.html
article.css
article.js // contains helpers, events and onCreated/onRendered
Then I have to import the files in startup/client (-> all ui files of this module AND all api files) and startup/server (-> just all api files)...
Right?
A few points:
You've put everything under imports/api. That directory is designed for collections, methods, helpers, 'business logic' and public API (e.g. if you expose a REST API, you'd do it from within that directory). Use imports/ui for your templates (including their styles and associated .js files).
You don't need to differentiate between client and server directories within imports. Just import the files you need from the respective main entry points (i.e. client/main.js and server/main.js). This point is a little more complex than I suggest here, see the link to 'structure' in the Meteor Guide, below.
index.js doesn't seem like a logical place to put your Articles collection. I'd make a file at /imports/api/articles/articles.js for it. See http://guide.meteor.com/structure.html for a good overview about where to put things and why.
Also, in the interests of following best-practices, use a default export for your Articles collection: http://guide.meteor.com/code-style.html#collections
To answer your question about how much of the file is exported (i.e. which functions), there's not much you can do about everything being loaded. The bundler needs to read the entire JS file anyway (imagine you exported an object and then changed it further down in the same file– not the best practice, but possible). If you're not using a function though, by all means, don't import it! And you can always split up your methods into seperate files if they get unmanageable.
Regarding your question about only importing bits for certain user roles: always avoid using imports or other types of obfuscation for security. The ideal way to do security on Meteor is to assume ANYTHING is accessible on the client (it pretty much is) and code your server-side code accordingly. That means, if you have an admin area, assume that anyone can access it. You can do checks in server methods and publications for this.userId and do a database lookup there to ensure the user has the correct privileges. Again, the guide has more info about this: http://guide.meteor.com/security.html
A final note about imports/exports: the idea behind them is not to reduce code size, but to provide a graph of what is actually being used (and leaving out the files that aren't) to make hot code reloading faster for a better development experience. They also make for cleaner application code that is easier to understand, because you don't have random magical globals swimming around that could have come from anywhere, and help to keep logically distinct pieces of code separate.
Best of luck :)

Meteor object available in Console, but throws 'cannot read property findOne of undefined

I'm new to Meteor and I've been stuck on this problem for hours.
I've created a collection in a file 'sessions.js' at imports/lib/sessions.js
import { Meteor } from 'meteor/meteor';
import { Mongo } from 'meteor/mongo';
Sessions = new Mongo.Collection('sessions');
I've created a few documents in the database using the browser console. I then want to return one document using the template helper:
import { Template } from 'meteor/templating';
import { Sessions } from '../../../../imports/lib/sessions.js';
Template.CarCamping.helpers({
test: function() {
return Sessions.findOne({name: 'David'});
}
});
This throws the following error:
debug.js:41Exception in template helper: TypeError: Cannot read property 'findOne' of undefined
at Object.test (http://localhost:3000/app/app.js?hash=e301d24dd47ccd4c73ae01a856896ad67acaaca0:77:32)
at http://localhost:3000/packages/blaze.js?hash=ef41aed769a8945fc99ac4954e8c9ec157a88cea:2994:16
at http://localhost:3000/packages/blaze.js?hash=ef41aed769a8945fc99ac4954e8c9ec157a88cea:1653:16
at http://localhost:3000/packages/blaze.js?hash=ef41aed769a8945fc99ac4954e8c9ec157a88cea:3046:66
at Function.Template._withTemplateInstanceFunc (http://localhost:3000/packages/blaze.js?hash=ef41aed769a8945fc99ac4954e8c9ec157a88cea:3687:12)
at http://localhost:3000/packages/blaze.js?hash=ef41aed769a8945fc99ac4954e8c9ec157a88cea:3045:27
at Spacebars.call (http://localhost:3000/packages/spacebars.js?hash=65db8b6a8e3fca189b416de702967b1cb83d57d5:172:18)
at Spacebars.mustacheImpl (http://localhost:3000/packages/spacebars.js?hash=65db8b6a8e3fca189b416de702967b1cb83d57d5:106:25)
at Object.Spacebars.mustache (http://localhost:3000/packages/spacebars.js?hash=65db8b6a8e3fca189b416de702967b1cb83d57d5:110:39)
at ._render (http://localhost:3000/app/app.js?hash=e301d24dd47ccd4c73ae01a856896ad67acaaca0:30:22)
What I've tried already:
Made sure autopublish is on, so it's not a problem in subscribing or publishing data.
Made sure creating collection isn't only done on server. The path 'imports/lib/sessions.js' is accessible for both client and server if I understood correctly.
The html contains the template tags with template name
The thing is: When i query the db from the console using 'Sessions.findOne({name: 'David'});' it does give me the expected result. But from the code itself it's not working.
Help is much appreciated! Thanks in advance.
You are not exporting Sessions from imports/lib/sessions.js, so when you go to import it, Sessions won't be defined. Try this:
import { Mongo } from 'meteor/mongo';
export const Sessions = new Mongo.Collection('sessions');
You can see a similar pattern with the Todos collection here.

Categories

Resources