Develop Reference

javascript cannot convert undefined or null to object question

I am trying javascript for the first time and I am having this trouble with the example:
https://www.twilio.com/blog/web-scraping-and-parsing-html-with-node-js-and-cheerio
It is a web scrapper example that uses got and cheerio, both of which I have installed. But when i run the sample code it gives me 'cannot convert undefined or null to object error.
Why is that? I didn't change anything from the example at all.
the code in question:
const $ = cheerio.load(response.body);
$('a').each((i, link) => {
const href = link.attribs.href;
console.log(href);
});
}).catch(err => {
console.log(err);
});

How does your index.js file look like? I did the tutorial and my code is working. Maybe you are miss typed the url?
Here is my index.js
const fs = require("fs");
const cheerio = require("cheerio");
const got = require("got");
const vgmUrl = "https://www.vgmusic.com/music/console/nintendo/nes";
got(vgmUrl)
.then((response) => {
const $ = cheerio.load(response.body);
$("a").each((i, link) => {
const href = link.attribs.href;
console.log(href);
});
})
.catch((err) => {
console.log(err);
});

How do I properly route data through my Node API?

I have the following files:
My routes - where the orders_count route lives:
routes/index.js
const express = require('express');
const router = express.Router();
const transactionsController = require('../controllers/transactionsController');
const ordersController = require('../controllers/ordersController');
const ordersCountController = require('../controllers/ordersCountController');
router.get('/transactions', transactionsController);
router.get('/orders', ordersController);
router.get('/orders_count', ordersCountController);
module.exports = router;
I then have my orders count controller living in the controllers directory:
controllers/ordersCountController.js
const ordersCountService = require('../services/ordersCountService');
const ordersCountController = (req, res) => {
ordersCountService((error, data) => {
if (error) {
return res.send({ error });
}
res.send({ data })
});
};
module.exports = ordersCountController;
My controller then calls my order count service which fetches data from another API.
services/ordersService.js
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = (req, res) => {
const url = ...;
const settings = { method: 'Get'};
fetch(url, settings)
.then(res => {
if (res.ok) {
res.json().then((data) => {
return data;
});
} else {
throw 'Unable to retrieve data';
}
}).catch(error => {
console.log(error);
});
}
module.exports = ordersCountService;
I'm trying to return the JSON response. I initially had it setup with requests but looking at the NPM site, it appears that it's depreciated so have been digging through how to use node-fetch.
I have tried both 'return data' and res.send({data}), but neither are solving the problem.
I am still new to this so I am likely missing something very obvious, but how come I am not sending the JSON back through so that it displays at the /api/orders_count endpoint?
I keep thinking I messed something up in my controller but have been looking at it for so long and can't seem to figure it out.
Any help would be greatly appreciated and if there is anything I can add for clarity, please don't hesitate to ask.
Best.

please learn promises and await syntax. life will be easier.
never throw a string. always prefer a real error object, like that : throw new Error('xxx'); that way you will always get a stack. its way easier to debug.
avoid the callback hell : http://callbackhell.com/
you need to decide if you want to catch the error in the controller or in the service. no need to do in both.
in the controller you call the service that way :
ordersCountService((error, data) => {
but you declare it like that :
const ordersCountService = (req, res) => {
which is not compatible. it should look like this if you work with callback style :
const ordersCountService = (callback) => {
...
if (error) return callback(error)
...
callback(null, gooddata);
here is an example to flatten your ordersCountService function to await syntax, which allows the "return data" you were trying to do :
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = async (req, res) => {
const url = ...;
const settings = { method: 'Get'};
try {
const res = await fetch(url, settings);
if (!res.ok) throw new Error('Unable to retrieve data');
return await res.json();
} catch(error) {
console.log(error);
}
}
module.exports = ordersCountService;
in fact i would prefer to error handle in the controller. then this woud be sufficient as a service
const fetch = require('node-fetch');
// connect to api and make initial call
const ordersCountService = async () => {
const url = ...;
const settings = { method: 'Get'};
const res = await fetch(url, settings);
if (!res.ok) throw new Error('Unable to retrieve data');
return await res.json();
}
module.exports = ordersCountService;
then you can call this funtion like this :
try {
const data = await ordersCountService(req, res);
} catch(err) {
console.log(err);
}
//or
ordersCountService(req, res).then((data) => console.log(data)).catch((err) => console.error(err));

Cloud Function to export new documents from Firestore to GCP bucket

I am trying to write a cloud function to export only the new documents getting added to my 'reviews' sub-collection. The trigger for this cloud function is: Cloud Firestore. However, my cloud function deployment fails through the console. Could someone please help me understand what's wrong with my cloud function?
Error message:
Deployment failure:
Build failed: /workspace/index.js:26
}
^
SyntaxError: missing ) after argument list
at new Script (vm.js:83:7)
at checkScriptSyntax (internal/bootstrap/node.js:620:5)
at startup (internal/bootstrap/node.js:280:11)
at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3); Error ID: d984e68f
Cloud function code:
const firestore = require('#google-cloud/firestore');
const client = new firestore.v1.FirestoreAdminClient();
const bucket = 'gs://bucket_name'
exports.scheduledFirestoreBackup = (event, context) => {
const databaseName = client.databasePath(
// process.env.GCLOUD_PROJECT,
"fs124",
'(default)'
);
return client
.exportDocuments({
name: databaseName,
outputUriPrefix: bucket,
collectionIds: ['reviews'],
})
.onSnapshot()
.then(snap => {
snap.forEach(doc => {
const response = doc.data();
console.log(doc.data());
return response;
}
});
Console snippet:

The message you are getting, SyntaxError: missing ) after argument list is pretty clear. You are missing the closing curly bracket} and parenthesis) of then(). It should look something like this:
const firestore = require('#google-cloud/firestore');
const client = new firestore.v1.FirestoreAdminClient();
const bucket = 'gs://bucket_name'
exports.scheduledFirestoreBackup = (event, context) => {
const databaseName = client.databasePath(
// process.env.GCLOUD_PROJECT,
"fs124",
'(default)'
);
return client
.exportDocuments({
name: databaseName,
outputUriPrefix: bucket,
collectionIds: ['reviews'],
})
.onSnapshot()
.then(snap => {
snap.forEach(doc => {
const response = doc.data();
console.log(doc.data());
return response;
});
});
};

How do I seed mongodb with data from an external API?

I'm trying to learn NodeJS. I'm using mongoose & mLab. I'm new to every one of these technologies.
My model at the moment looks like this. I will add a few things to the schema later.
const mongoose = require("mongoose");
const fetchData = require("../seed");
const schema = mongoose.Schema;
const dataSchema = new Schema({});
module.exports = recallData = mongoose.model("recalls", dataSchema);
I also made a seed file for fetching data..
const Recall = require("./models/Recall");
module.exports = function getData(req, res) {
const urls = [url1, url2, url3];
urls.map(url => {
fetch(url)
.then(res => res.json())
.then(data =>
data.results.map(recalls => {
let recs = new Recall(recalls);
recs.save;
})
);
});
}
my question is how do I make the fetch run and populate the database? Is there a command or a mongoose function that will do that?
I know that I'm basically trying to emulate Rails with a seed file. Maybe it's not the way to do it in Node. Any help is super appreciated.

Turns out it's pretty simple. All I needed was a nights sleep. I needed to connect to mongoose and after save(), disconnect.
Now the code looks like this. I still need to add and edit some stuffs in it. Any smart refactoring advice is appreciated.
const mongoose = require("mongoose");
const Recall = require("./models/Recall");
const db = require("./config/keys").mongoURI;
const fetch = require("node-fetch");
const URLS = require("./config/seedURLs");
let resultData;
let saveCounter = 0;
mongoose
.connect(db)
.then(() => console.log("mongodb connection success"))
.catch(error => console.log(error));
URLS.map(async url => {
try {
const response = await fetch(url);
const json = await response.json();
resultData = [...json.results];
for (let i = 0; i < resultData.length; i++) {
let temp = new Recall({
key1: resultData[i].key1,
key2: resultData[i].key2,
.
.
.
});
temp.save(() => {
saveCounter++;
if (saveCounter === resultData.length) {
mongoose
.disconnect()
.then(() => console.log("mongodb disconnected"))
.catch(error => console.log(error));
}
});
}
} catch (error) {
console.log(error);
}
});
Run node seed.js command.
This is the general idea.

Seed db with sequelize seed files before running tests with

How do I do rake style commands in my test file (Jest) with sequelize seeder files?
I'm trying to do the same thing as this, but with sequelize.
describe('routes : movies', () => {
beforeEach(() => {
return knex.migrate.rollback()
.then(() => { return knex.migrate.latest(); })
.then(() => { return knex.seed.run(); });
});
afterEach(() => {
return knex.migrate.rollback();
});
});

I know I'm late to answer this but I just spent the past week trying to solve this issue. I have been able to successfully do this using Sequelize in conjunction with their sister project, Umzug. You will have to read the documentation for your specific issue but I can copy my test file so you can get an idea of how I did it. I'm happy to help someone if they still struggle with it after looking at the files.
// account.test.js
const models = require('../models/index.js');
const migrations = require("../index");
beforeAll(async () => {
await migrations.up().then(function() {
console.log("Migration and seeding completed")
});
});
afterAll( async () => {
await migrations.down().then(function() {
console.log("Migrations and down seeding completed");
})
const users = await models.User.findAll();
expect(users).toStrictEqual([]);
});
describe("Integration Test", () => {
it("Account integration test", async () => {
const data = { userId: 1210}
const users = await models.User.findAll();
console.log("All users:", JSON.stringify(users, null, 2));
expect(users[0].firstName).toBe('John');
expect(data).toHaveProperty('userId');
});
});
My index.js file
// index.js
const config = require('/config/config.json');
const { Sequelize } = require('sequelize');
const { Umzug, SequelizeStorage } = require('umzug');
const sequelize = new Sequelize(config);
const umzugMigration = new Umzug({
migrations: { glob: 'migrations/*.js' },
context: sequelize.getQueryInterface(),
storage: new SequelizeStorage({ sequelize }),
logger: console,
});
const umzugSeeding = new Umzug({
migrations: { glob: 'seeders/*.js' },
context: sequelize.getQueryInterface(),
storage: new SequelizeStorage({ sequelize }),
logger: console,
});
module.exports.up = () => umzugMigration.up().then(() => umzugSeeding.up());
module.exports.down = () => umzugSeeding.down();

I think you shouldn't make real DB requests while testing your code. Mock your DB request and return the data set from your mock function if it's needed. Otherwise, it looks like you testing a library, in your case this lib is knex.
Read for more details regarding mocks https://jestjs.io/docs/en/mock-functions