I'm trying to write a mongo script to import a jsonArray from a JSON file. My script is in .js format and I execute it with load() command in mongo shell. Is it possible to do it with a mongo script?
I know I can use mongoimport instead. But I want to know a way to do it with a script.
The contents of my current script in which the import part is missing is given below..
var db = connect("localhost:27017/fypgui");
//Import json to "crimes" collection here
var crimes = db.crimes.find();
while (crimes.hasNext()){
var item = crimes.next();
var year =(item.crime_date != null)?(new Date(item.crime_date)).getFullYear():null;
db.crimes.update( {_id: item._id}, {$set: {crime_year: year}});
}
There is another answer to this question. Even though it's a bit old I am going to respond.
It is possible to do this with the mongo shell.
You can convert your JSON to valid JavaScript by prefixing it with var myData= then use the load() command to load the JavaScript. After the load() you will be able to access your data from within your mongo script via the myData object.
data.js
var myData=
[
{
"letter" : "A"
},
{
"letter" : "B"
},
{
"letter" : "C"
}
]
read.js
#!/usr/bin/mongo --quiet
// read data
load('data.js');
// display letters
for (i in myData) {
var doc = myData[i];
print(doc.letter);
}
For writing JSON is easiest to just load your result into a single object. Initialize the object at the beginning with var result={} and then use printjson() at the end to output. Use standard redirection to output the data to a file.
write.js
#!/usr/bin/mongo --quiet
var result=[];
// read data from collection etc...
for (var i=65; i<91; i++) {
result.push({letter: String.fromCharCode(i)});
}
// output
print("var myData=");
printjson(result);
The shebang lines (#!) will work on a Unix type operating system (Linux or MacOs) they should also work on Windows with Cygwin.
It is possible to get file content as text using undocumented cat() function from the mongo shell:
var text = cat(filename);
If you are interested cat() and other undocumented utilities such as writeFile are defined in this file: shell_utils_extended.cpp
Having file's content you can modify it as you wish or directly pass it to JSON.parse to get JavaScript object:
jsObj = JSON.parse(text);
But be careful: unfortunately JSON.parse is not an equivalent of mongoimport tool in the sense of its JSON parsing abilities.
mongoimport is able to parse Mongo's extended JSON in canonical format. (Canonical format files are created by bsondump and mongodump for example. For more info on JSON formats see MongoDB extended JSON).
JSON.parse does not support canonical JSON format. It will read canonical format input and will return JavaScript object but extended data type info present in canonical format JSON will be ignored.
No, the mongo shell doesn't have the capability to read and write from files like a fully-fledged programming environment. Use mongoimport, or write the script in a language with an official driver. Node.js will have syntax very close to the mongo shell, although Node.js is an async/event-driven programming environment. Python/PyMongo will be similar and easy to learn if you don't want to deal with structuring the logic to use callbacks.
Hey I know this is not relevent but, every time I need to import some jsons to my mongo db I do some shity copy, paste and run, until I had enough!!!
If you suffer the same I v written a tiny batch script that does that for me. Intrested?
https://github.com/aminjellali/batch/blob/master/mongoImporter.bat
#echo off
title = Mongo Data Base importing tool
goto :main
:import_collection
echo importing %~2
set file_name=%~2
set removed_json=%file_name:.json=%
mongoimport --db %~1 --collection %removed_json% --file %~2
goto :eof
:loop_over_files_in_current_dir
for /f %%c in ('dir /b *.json') do call :import_collection %~1 %%c
goto :eof
:main
IF [%1]==[] (
ECHO FATAL ERROR: Please specify a data base name
goto :eof
) ELSE (
ECHO #author amin.jellali
ECHO #email a.j.amin.jellali#gmail.com
echo starting import...
call :loop_over_files_in_current_dir %~1
echo import done...
echo hope you enjoyed me
)
goto :eof
Related
I'm trying to read an updating JSON file from syslog-ng. Currently, syslog, a logging software, is set to continually append a JSON file with logs of the data I want. I'm displaying the data on my cyber attack map only for only 30 seconds until it's not needed anymore. I can read the file and parse what I need, but is there a way to, over time, read & parse only the most recent additions to the file?
Sample code:
//Assume JSON output = {attack source, attack destination, attack type}
//Required modules
var JSONStream = require('JSONStream')
var fs = require('fs');
//Creates readable stream for JSON file parsing
var stream = fs.createReadStream( 'output.json', 'utf8'),
parser = JSONStream.parse(['source', 'dest', 'type']);
//Send read data to parser function
stream.pipe(parser);
//Intake data from parser function
parser.on('data', function (obj) {
//Do something with the object
console.log(obj);
});
I'm using JSONStream to avoid having to read the whole log file into memory, JSONstream should still be able to parse the bits I want, but is there a method to only read changes after the original reading is complete?
Use this code example provided in the library
JSONStream Test code
You don't have to wait for the end, you can use the callback to do your work object by object
But the file structure should suite the library expectation as the files given in the folder
Example file all_npm.json
Lets say, my file has following content and situated at /home/usr1/Documents/companyNames.txt
Name1
Name 2
Name 3
Millions of names...
I tried the following code:
$> var string = cat('home/usr1/Documents/companyNames.txt');
$> string = string.split('\n');
$> db.records.find({field: {$in: string}});
As per the code in link Can I read a csv file inside of a Mongo Shell Javascript file?
This works if the file is small in size, but when the file has millions of lines it fails. The whole lines in the file is trying to fit inside the memory and gets crashed. Is there any other way to process big files inside Mongo shell with Java script?
Mongo isn't very good with larges queries.
You might have to go the Javascript way :
var string = cat('home/usr1/Documents/companyNames.txt');
string = string.split('\n');
let results = [];
string.forEach(string => result.push(db.records.find({field: {$eq: string}})));
Is it possible to read/parse an exported Javascript or Typescript JSON-like object in Python?
For example:
In myJava.js:
export const myObj = {
entry: val
entry2: val2
...
}
In parseJava.py:
def parseJava():
# Some code to read in the javascript object
javaObjAsDictionary = someFunction("myJava.js")
Does such a "someFunction()" exist? If not, are there any clean ways around this?
Thanks in advance!
If you want to parse JSON, there is a library called JSON in Python 2 and 3 that does the encoding and decoding.
Specifically, you can use the json.loads or json.load methods to get a Python object (dictionary/list) from your JSON.
Something like:
import json
jsonDict = json.load("myJSON.json")
I am not sure exactly if this is what you need, hope this gives you a start.
# following code assumes that 'data.json' file exists in the current working directory
with open('data.json', 'r') as jsonFile: # opens JSON file in read only mode
# loads the content of JSON file and converts it into python dictionary object
dictionary = json.load(jsonFile)
print(dictionary)
I know that it's possible to read and get data of JSON file, but I didn't find any information on how to write an object to JSON file using jQuery. I understand some jQuery, but I dont have any idea how I could do this.
This is the structure of my JSON file:
{
"1": [
"6-5-2015",
"7-5-2015",
"10-5-2015"
]
}
This is an object variable that I want to write into JSON file:
var object = {"2": ["9-5-2015", "14-5-2015", "22-5-2015"]};
How can I push or insert this object to the end of my JSON file and save it, so that the JSON file could look like this?
{
"1": [
"6-5-2015",
"7-5-2015",
"10-5-2015"
],
"2": [
"9-5-2015",
"14-5-2015",
"22-5-2015"
]
}
You cannot write a file locally with Javascript, that would be a major security concern. I suggest you to move the file to your server and do a Public Api where you send the new content and write it server-side. Then request by GET the file in order to read it. Remember that you will have to lock and release the file accordingly in order to avoid loosing changes between requests.
You can't.
JavaScript does not access your disc so it can't directly write into file, so You should have some server side logic.
Reason why it can read that file because on your dist location of that file is URL. But URL on Your drive.
You cannot write a file locally but you can save cookies locally.
For managing cookies with JS you can use a plugin available here..
https://github.com/carhartl/jquery-cookie
//set
var object = {"2": ["9-5-2015", "14-5-2015", "22-5-2015"]};
$.cookie("mydata", object );
....
//get
var object = jQuery.parseJSON($.cookie('mydata'));
I use some Python scripts on server-side to compile JavaScript files to one file and also add information about these files into database.
For adding the information about the scripts I now have yaml file for each js file with some info inside it looking like this:
title: Script title
alias: script_alias
I would like to throw away yaml that looks redundant here to me if I can read these variables directly from JavaScript that I could place in the very beginning of the file like this:
var title = "Script title";
var alias = "script_alias";
Is it possible to read these variables with Python easily?
Assuming you only want the two lines, and they are at the top of the file...
import re
js = open("yourfile.js", "r").readlines()[:2]
matcher_rex = re.compile(r'^var\s+(?P<varname>\w+)\s+=\s+"(?P<varvalue>[\w\s]+)";?$')
for line in js:
matches = matcher_rex.match(line)
if matches:
name, value = matches.groups()
print name, value
Have you tried storing the variables in a JSON format? Then, both javascript and python can easily parse the data and get the variables.