I am using docker to containerize a Flask API. The Flask API is accessing MongoDB using the "links" keyword in the docker-compose.yml file shown below:
app:
build: .
command: python -u app.py
ports:
- "5000:5000"
volumes:
- .:/app
#Linking db.
links:
- db
db:
image: mongo:latest
hostname: test_mongodb
environment:
- MONGO_INITDB_DATABASE=Exploit_Resources
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=pass
# Data provider.
volumes:
- ./init-db.js:/docker-entrypoint-initdb.d/init-db.js:ro
ports:
- 27017:27017
The link is working fine, and the file given under volumes (init-db.js) is supposed to provide the Mongo container with data. Here is my init-db.js file:
db = db.getSiblingDB("Exploit_Resources");
db.EDB.drop();
db.EDB.insertMany([
// Insert data here.
]);
The data I want to provide MongoDB comes from a remote csv file. I tried using PapaParse to access and feed the data, but I was having some difficulties importing it.
I tried this code inside the insertMany method:
const {StringStream} = require("scramjet");
const request = require("request");
request.get("https://srv.example.com/main.csv") // fetch csv
.pipe(new StringStream()) // pass to stream
.CSVParse() // parse into objects
.consume(object => console.log("Row:", object)) // do whatever you like with the objects
.then(() => console.log("all done"))
I got the code from here. It did not work because a ReferenceError was thrown, saying that 'require' is not defined.
Is there a way to fix this code or another way to get data from a remote csv file and provide it to the MongoDB container from the init-db.js?
Related
As I understand, async fetch() in Nuxt makes server side call before render which I'm trying to use to render post data but I don't understand why <NuxtLink> and <a href... are working differently.
<NuxtLink> works to render content if url in async fetch /api/readpost/.. is used but it renders before finishing fetching /api/readpost, so the error 404 post flashes before updating after the data shows up
<a href... works correctly if url in async fetch is http://spserver:3002/api/readpost/... and correctly waits for the fetch before rendering
Client and server request urls would be different because the project is running in docker behind nginx. My concern here is to always make sure the data shows up before rendering happens and if possible make <NuxtLink> behave the same way regular <a href.. seems to be working.
So, what is causing the difference, how would I ensure async fetch always makes server side request and waits for data before rendering?
Video demo:
On page reload I get console errors like this:
Error: connect ECONNREFUSED 127.0.0.1:80
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 80,
config: {
url: '/api/readpost/23',
method: 'get',
headers: {
...
Code:
I have something like this in a nuxt page to render post data:
<template>
<template v-if="posts && posts.length">
<PostRender :post="posts[0]" />
</template>
<div v-else-if="!$fetchState.pending">
404 POST NOT FOUND, PLEASE DOUBLE CHECK THE URL
</div>
</template>
<script>
export default {
data() {
return {
postid : this.$route.params.post,
posts : null,
}
},
async fetch()
{
let res = await this.$axios({
method : 'GET',
url : '/api/readpost/' + this.postid
});
this.posts = res.data;
}
</script>
and this lists the posts and links
<template>
<div>
<div v-for="p in posts">
<NuxtLink :to="'/post/' + p.id">NuxtLink</NuxtLink>
<hr>
</div>
</div>
</template>
<script>
export default {
data() {
return {
posts : null
}
},
mounted() {
this.$axios({
method : 'GET',
url : '/api/readlatest'
})
.then((res) =>
{
this.posts = res.data;
})
.catch((err) =>
{
console.error(err);
});
}
}
</script>
And my docker compose looks something like this
version : "3"
services:
spserver:
build: .
container_name: spserver
working_dir: /app
volumes:
- ./server/:/app
ports:
- 7002:3002
entrypoint: npm run dev
spclient:
image: node:14.18.1-alpine
container_name: spclient
working_dir: /app
volumes:
- ./client/:/app
environment:
- PORT=3001
- HOST=0.0.0.0
- "BASE_URL=/"
entrypoint: sh -c "npm run build && npm run dev"
depends_on:
- spserver
spnginx:
image: nginx:1.18-alpine
container_name: spnginx
volumes:
- ./nginx.conf/:/etc/nginx/nginx.conf
ports:
- 9001:80
depends_on:
- spclient
- spserver
Relavant bit of nuxt.config.js:
axios: {
baseURL: process.env.BASE_URL
},
Versions:
"nuxt": "^2.15.8",
"#nuxtjs/axios": "^5.13.6",
This solved my issue, although a little confused about the whole thing still:
url : (process.server ? 'http://spserver:3002/api/readpost/' : '/api/readpost/') + postid
Nuxt is isomorphic but will make a call to the server only on the initial render. That one will be generated on the server (supposing ssr: true) and will be sent to your browser.
There, you will only get client-side navigation on any successful route changes thanks to Vue router.
If you use an a tag, you will be nuking your whole Nuxt/Vue app and creating a brand new from scratch, hence have another backend generation (clearly not the way to go as you can guess).
If you want to have an example of data fetching using fetch() (lifecycle hook), you can check this answer. It can also be done in a blocking way by using the asyncData().
PS: I recommend you using the ones above rather than created() and mounted() in a Nuxt app + try to use async/await all the time rather than a mix between those + .then.
Preamble
To start off, I'm not a developer; I'm just an analyst / product owner with time on their hands. While my team's actual developers have been busy finishing off projects before year-end I've been attempting to put together a very basic API server in Node.js for something we will look at next year.
I used Swagger to build an API spec and then used the Swagger code generator to get a basic Node.js server. The full code is near the bottom of this question.
The Problem
I'm coming across an issue when writing out to a log file using the fs module. I know that the ENOENT error is usually down to just specifying a path incorrectly, but the behaviour doesn't occur when I comment out the Swagger portion of the automatically generated code. (I took the logging code directly out of another tool I built in Node.js, so I'm fairly confident in that portion at least...)
When executing npm start, a few debugging items write to the console:
"Node Server Starting......
Current Directory:/mnt/c/Users/USER/Repositories/PROJECT/api
Trying to log data now!
Mock mode: disabled
PostgreSQL Pool created successfully
Your server is listening on port 3100 (http://localhost:3100)
Swagger-ui is available on http://localhost:3100/docs"
but then fs throws an ENOENT error:
events.js:174
throw er; // Unhandled 'error' event
^
Error: ENOENT: no such file or directory, open '../logs/logEvents2021-12-24.log'
Emitted 'error' event at:
at lazyFs.open (internal/fs/streams.js:277:12)
at FSReqWrap.args [as oncomplete] (fs.js:140:20)
Investigating
Now normally, from what I understand, this would just mean I've got the paths wrong. However, the file has actually been created and the first line of the log file has been written just fine
My next thought was that I must've set the fs flags incorrectly, but it was set to 'a' for append:
var logsFile = fs.createWriteStream(__logdir+"/logEvents"+dateNow()+'.log',{flags: 'a'},(err) =>{
console.error('Could not write new Log File to location: %s \nWith error description: %s',__logdir, err);
});
Removing Swagger Code
Now here's the weird bit: if I remove the Swagger code, the log files write out just fine and I don't get the fs exception!
This is the specific Swagger code:
// swaggerRouter configuration
var options = {
routing: {
controllers: path.join(__dirname, './controllers')
},
};
var expressAppConfig = oas3Tools.expressAppConfig(path.join(__dirname, '/api/openapi.yaml'), options);
var app = expressAppConfig.getApp();
// Initialize the Swagger middleware
http.createServer(app).listen(serverPort, function () {
console.info('Your server is listening on port %d (http://localhost:%d)', serverPort, serverPort);
console.info('Swagger-ui is available on http://localhost:%d/docs', serverPort);
}).on('error',console.error);
When I comment out this code, the log file writes out just fine.
The only thing I can think that might be happening is that somehow Swagger is modifying (?) the app's working directory so that fs no longer finds the same file?
Full Code
'use strict';
var path = require('path');
var fs = require('fs');
var http = require('http');
var oas3Tools = require('oas3-tools');
var serverPort = 3100;
// I am specifically tried using path.join that I found when investigating this issue, and referencing the app path, but to no avail
const __logdir = path.join(__dirname,'./logs');
//These are date and time functions I use to add timestamps to the logs
function dateNow(){
var dateNow = new Date().toISOString().slice(0,10).toString();
return dateNow
}
function rightNow(){
var timeNow = new Date().toTimeString().slice(0,8).toString();
return "["+timeNow+"] "
};
console.info("Node Server Starting......");
console.info("Current Directory: " + __dirname)
// Here I create the WriteStreams
var logsFile = fs.createWriteStream(__logdir+"/logEvents"+dateNow()+'.log',{flags: 'a'},(err) =>{
console.error('Could not write new Log File to location: %s \nWith error description: %s',__logdir, err);
});
var errorsFile = fs.createWriteStream(__logdir+"/errorEvents"+dateNow()+'.log',{flags: 'a'},(err) =>{
console.error('Could not write new Error Log File to location: %s \nWith error description: %s',__logdir, err);
});
// And create an additional console to write data out:
const Console = require('console').Console;
var logOut = new Console(logsFile,errorsFile);
console.info("Trying to log data now!") // Debugging logging
logOut.log("========== Server Startup Initiated ==========");
logOut.log(rightNow() + "Server Directory: "+ __dirname);
logOut.log(rightNow() + "Logs directory: "+__logdir);
// Here is the Swagger portion that seems to create the behaviour.
// It is unedited from the Swagger Code-Gen tool
// swaggerRouter configuration
var options = {
routing: {
controllers: path.join(__dirname, './controllers')
},
};
var expressAppConfig = oas3Tools.expressAppConfig(path.join(__dirname, '/api/openapi.yaml'), options);
var app = expressAppConfig.getApp();
// Initialize the Swagger middleware
http.createServer(app).listen(serverPort, function () {
console.info('Your server is listening on port %d (http://localhost:%d)', serverPort, serverPort);
console.info('Swagger-ui is available on http://localhost:%d/docs', serverPort);
}).on('error',console.error);
In case it helps, this is the project's file structure . I am running this project within a WSL instance in VSCode on Windows, same as I have with other projects using fs.
Is anyone able to help me understand why fs can write the first log line but then break once the Swagger code gets going? Have I done something incredibly stupid?
Appreciate the help, thanks!
Edit: Tried to fix broken images.
Found the problem with some help from a friend. The issue boiled down to a lack of understanding of how the Swagger module works in the background, so this will likely be eye-rollingly obvious to most, but keeping this post around in case anyone else comes across this down the line.
So it seems that as part of the Swagger initialisation, any scripts within the utils folder will also be executed. I would not have picked up on this if it wasn't pointed out to me that in the middle of the console output there was a reference to some PostgreSQL code, even though I had taken all reference to it out of the main index.js file.
That's when I realised that the error wasn't actually being generated from the code posted above: it was being thrown from to that folder.
So I guess the answer is don't add stuff to the utils folder, but if you do, always add a bunch of console logging...
I'm trying to load an env variable with NextJS version 9.5.
I've created a file named .env.production at the root of my project following the official NextJS documentation over here: https://nextjs.org/docs/basic-features/environment-variables
Here's my env file content:
# APP VARIABLES
# ...
# Webservices
NEXT_PUBLIC_WEBSERVICE_HOST=$WEBSERVICE_HOST
# Keycloak
NEXT_PUBLIC_KEYCLOAK_HOST=$KEYCLOAK_HOST
NEXT_PUBLIC_KEYCLOAK_REALM=$KEYCLOAK_REALM
NEXT_PUBLIC_KEYCLOAK_CLIENT_ID=$KEYCLOAK_CLIENT_ID
I need these variables to be available on the browser as well so I've prepended the env variable name with NEXT_PUBLIC_.
Once I launch the app, I can see that the env file are loaded correctly:
> project# start /app
> NODE_ENV=production node dist/server/server.js
info - Loaded env from /app/.env.production
> Config - API host: https://<redacted>
> Config - Keycloak host: https://<redacted>
> Ready on localhost:3000 - env production
But once in the browser when I try to call the API using this piece of code
const keycloakUrl = process.env.NEXT_PUBLIC_KEYCLOAK_HOST as string
const keycloakRealm = process.env.NEXT_PUBLIC_KEYCLOAK_REALM as string
export const clientId = process.env.NEXT_PUBLIC_KEYCLOAK_CLIENT_ID as string
const instance = axios.create({
baseURL: `${keycloakUrl}/auth/realms/${keycloakRealm}`,
timeout: 30000,
})
# ...
instance.post('/').catch(console.err);
I can see that my variables keycloakUrl and keycloakRealm are empty when I call my API.
How can I solve this problem ?
Dear Strongloop community
I am following the example provided by loopback for evaluating local-storage data store.
It works fine and I can use the rest API to create and retrieve files from the local file system (used as a data source).
I had to used "$ npm install loopback-component-storage --save".
the datasource defined in datasource.json is:
{
"db": {
"name": "db",
"connector": "memory"
},
"localFileSystemStorage": {
"name": "localFileSystemStorage",
"connector": "loopback-component-storage",
"provider": "filesystem",
"root": "./property_data"
}
}
the ./property_data directory is where all the data is stored. (relative to the project directory) and I can access it through rest with no issues.
Next I was trying to generate Angular service component so I can connect to the loopback server using Angular application. So inside the client directory I executed:
../example-2.0/client$ lb-ng ../server/server.js js/lb-services.js
and it failed:
/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/providers/filesystem/index.js:25
throw new Error('FileSystemProvider: Path does not exist: ' + this.root);
^
Error: FileSystemProvider: Path does not exist: ./server/storage
at new FileSystemProvider (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/providers/filesystem/index.js:25:11)
at Object.module.exports.createClient (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/providers/filesystem/index.js:17:10)
at Object.createClient (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/factory.js:86:18)
at new StorageService (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/storage-service.js:29:25)
at Object.exports.initialize (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-component-storage/lib/storage-connector.js:12:19)
at DataSource.setup (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-datasource-juggler/lib/datasource.js:332:17)
at new DataSource (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback-datasource-juggler/lib/datasource.js:109:8)
at Registry.createDataSource (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback/lib/registry.js:349:12)
at dataSourcesFromConfig (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback/lib/application.js:415:19)
at EventEmitter.app.dataSource (/home/eval/projects/loopback-component-storage/example-2.0/node_modules/loopback/lib/application.js:222:12)
Am I missing something?
I could really use some help here.
Thanks
In datasource.json file you have provided the relative path ./property_data with respect to /server folder.
Thus you need to call any method from that folder only.
Now when you are running it through the server then you are running it from /server folder thats why it runs without an issue but for angular Sdk generation you are doing it from client/ folder. Thus its genetating Path does not exist error.
Thus the correct syntax would be:
../example-2.0/server$ lb-ng server.js ../client/js/lb-services.js
I have been successfully executing javascripts from the mongo shell
We have now enabled authorization on the mongo database
The permissions have been given and if I login to the shell and execute queries everything is fine
However, I am trying to execute the javascript from command line and I have a problem
I am issuing the command
mongo -u -p -authenticationDatabase admin GetProcessDate.js
I get the error message
MongoDB shell version: 2.6.5
connecting to: test
2014-10-15T06:44:11.451-0700 error: { "$err" : "not authorized for query on bvmaster.ProcessDate", "code" : 13 } at src/mongo/shell/query.js:131
failed to load: GetProcessDate.js
When I disable authorization this javascript executes just fine
mongo GetProcessDate.js
When I enable authorization and login to the shell everything works fine
mongo -u <user>-p <password> -authenticationDatabase admin
The javascript is a very simple one
mongo = new Mongo();
bvmasternew = mongo.getDB( "bvmaster" );
ProcessDateCursor = bvmasternew.ProcessDate.find();
ProcessDateRec = ProcessDateCursor.next();
print( ProcessDateRec.Date);
Any help will be appreciated!!
After a lot of searchs I found the following;
When you are executing a javascript on the mongo command line, it does not matter what parameters you pass on the command line
inside your mongo javascript
you need to call the mongo with proper parameters
mongo = new Mongo( server:port );
admin = mongo.getDB( "admin" );
admin.auth( User, Password );
These lines basically authorizes the user to perform various operations in that mongo instance you have connected using server/port combination