For some background, I have a number of enterprise systems management products that run a REST API called Redfish. I can visit the root folder on one such device by going here:
https://ip.of.device/redfish/v1
If I browse the raw JSON from a GET request, the output includes JSON data that look like this. I'm truncating this due to how long it is, so perhaps some JSON syntax errors here.
{
Description: "Data",
AccountService: {
#odata.id: "/redfish/v1/AccountService"
},
CertificateService: {
#odata.id: "/redfish/v1/CertificateService"
}
}
Perhaps in my searching I'm using the wrong terminology, but each of those #odata.id items is basically a 'folder' I can navigate into. Each folder has additional data values, but still more folders. I can capture contents of folders I know about via javascript and parse the JSON simple enough, but there are hundreds of folders here, some multiple layers deep and from one device to the next, sometimes the folders are different.
Due to the size and dynamic nature of this data, is there a way to either recursively query this from an API itself or recursively 'scrape' an API's #odata.id 'folder' structure like this using Javascript? I'm tempted to write a bunch of nested queries in foreach loops, but there's surely a better way to accomplish this.
My eventual goal is to perform this from nodejs, parse the data, then present the data in a web form for a user to select what fields to keep, which we'll store for faster lookups in a mongodb database along with the path to the original data for more targeted api queries later.
Thanks in advance for any suggestions.
Related
I have recently begun to work with NodeJS, after a long time working with PHP, and i'm wondering if there was anything similar to 'echo'. a way of sending the data as parts, allowing me to send my own values in between. For example, writing the structure for a section of a website in html, then using node to send the relevant data inside of the div.
The short answer is "yes". You can repeatedly write to the response object and then call end on it when you are done.
response.setHeader("Content-Type", "text/plain");
response.write("foo");
response.write("bar");
response.write("baz");
response.end();
However, that generally isn't the approach taken for good reason.
Separating your concerns (e.g. splitting "data fetching" and "HTML generation") usually makes things more manageable.
This is why the most common way to build pages server-side with Node.js is to collect all the data for the page into an object and then pass it into a template.
You can, of course, split the data collection logic up into functions in their own modules so you can end up with something like:
const my_data = {
header: get_header_data(request),
content: get_template_name_data(request),
};
response.render('template_name', my_data)
With as much division into smaller chunks are you like.
I'm trying to create a key-value paired datastore in nodejs in which I'm trying to use a json file as a database. Now I'm trying to read a particular value of key and update it or delete it.
For eg.(dummy data)
{
name: 'my name',
no:12345,
course : {
name: 'cname',
id: 'cid'
}
}
Now I want to change it as
{
name: 'my name',
no:12345,
course : {
name: 'cname1',
id: 'cid1'
},
fees: {
primaryFee: 1000,
donation: 100,
}
}
Or even delete the course key with it's value.
One way I thought of to achieve this is read the entire file and store it in a variable(json parsed). And update the value in that variable and write the whole data to the file.
But this is not efficient as it reads and writes the whole data every time of an update.
So is there any efficient approach for this to update or delete a particular key in the file itself???
Yes, even in a high-level programming language like JS, it is possible to update parts of a file, though it is usually more common in low-level programming languages like e.g. C
For node.js, check-out the official documentation of the file system module, especially the write function: https://nodejs.org/api/fs.html#fs_fs_write_fd_buffer_offset_length_position_callback
Other possible solutions to your problem:
Use a database as eol suggested
Use an append-only file where you only append the updates. This is more efficient because not the whole file has to be written.
Split your database-file into several files (this is what databases often do below the surface)
Since you are working on a file based data store, you could use FS in node.js to read and write files. The readfilesync callback returns you the whole content of the file which should be parsed as JSON and changes are to be made.
So AFAIK the only solution is to read the whole file and then update or delete the value. In the worst case scenario you will traverse O(n) when you are trying to update or deleting the key.
You've discovered why databases are so complicated ;)
There isn't really a good way to update a json file without completely reading & parsing the data, editing and then writing the result again.
If you're looking for a simple file database, have a look at sqlite
For making these types of data changes, USE DB, for JSON files you have to store full data in the variable and perform the operations.
Is it possible to have a database under MySQL export rows into managed folders accessible to the web with the contents of each column be it's own text file OR json file?
To be honest, I can't find anywhere on how to do this?
My question seems to be quite niché.
Ill break down my problem.
lets say I have a database, with a table called "registry"
under registry, I would have 4 columns.
for example:
username nickname dob timestamp
bob2414 bobby 03211989 201019
sarah83 sars 10162002 231019
masterc carlo 07271997 261019
blahhbh umomi 03241999 281119
is there a way, upon a new entry, to trigger either a JS bot, or any kind of automation to create a structure on the web that could be called upon, for example.
domain.com/registry
bob2414/
nickname.txt
dob.txt
timestamp.txt
(or)
bob2414.json
sarah83/
nickname.txt
dob.txt
timestamp.txt
(or)
sarah83.json
masterc/
nickname.txt
dob.txt
timestamp.txt
(or)
masterc.json
and so on?
is there a page i can view, or any video explanation online?
Are there any better ways I can do this?
The application of my problem lies within creating these directories, as I don't want one huge json file hanging around, because i plan on these files being downloaded and accessed, and want smoother download times then managing a huge database.
The same would also apply to after I delete a row, would it manage to be deleted on its own?
please, and thank you. Ill help any way I can!
Answering my own question with a personal thought experiment.
In theory it would be easier to design an element in PHP to query the database with a GET request, and have data presented in JSON format, than to worry about managing files and directories with a java or javascript applet
I have a directory, with several sub-directories, holding a total of about a million JSON files. I need to import this entire thing into mongoDB. I imagine this is a very common problem, but I cannot find any tutorial for how to do so. Is there an easy solution here?
(Or should I write a script to iterate through the directories, read each file into a variable and then insert the content into my db?)
You're basically on the right path.
Iterate through the entire directory, read the files to create a JSON object in your code, and then just store your documents directly into MongoDB.
Take a look at this.
One big problem though: MongoDB stores your documents into Collections, with the condition that every document in your collection has the same general format/structure. Basically each document must have most, if not all of the same properties as each other. Otherwise you're going to have to use something different, or store all of the JSON as a property of an encapsulating document that you can then store.
Something like the following:
{
'_id': 'YOUR_DOCUMENT_ID',
'doc': 'JSON_OR_STRING_OF_YOUR_FILE'
}
I'm working on an AJAXy project (Dojo and Rails, if the particulars matter). There are several places where the user should be able to sort, group, and filter results. There are also places where a user fills out a short form and the resulting item gets added to a list on the same page.
The non-AJAXy implementation works fine -- the view layer server-side already knows how to render this stuff, so it can just do it again in a different order or with an extra element. This, however, adds lots of burden to the server.
So we switched to sending JSON from the server and doing lots of (re-)rendering client-side. The downside is that now we have duplicate code for rendering every page: once in Rails, which was built for this, and once in Dojo, which was not. The latter is basically just string concatenation.
So question part one: is there a good Javascript MVC framework we could use to make the rendering on the client-side more maintainable?
And question part two: is there a way to generate the client-side views in Javascript and the server-side views in ERB from the same template? I think that's what the Pragmatic Programmers would do.
Alternatively, question part three: am I completely missing another angle? Perhaps send JSON from the server but also include the HTML snippet as an attribute so the Javascript can do the filtering, sorting, etc. and then just insert the given fragment?
Well, every time you generate HTML snippets on the client and on the server you may end up with duplicated code. There is no good way around it generally. But you can do two things:
Generate everything on the server. Use AHAH when you need to generate HTML snippets dynamically. Basically you ask server to generate an HTML fragment, receive it asynchronously, and plug it in place using innerHTML or any other similar mechanism.
Generate everything on the client (AKA the thick client paradigm). In this case even for the initial rendering you pass data instead of pre-rendered HTML, and process the data client-side using JavaScript to make HTML. Depending on the situation you can use the data island technique, or request data asynchronously. Variant: include it as <script> using JSONP so the browser will make a request for you while loading the page.
Both approaches are very simple and have different set of pros and cons. Sometimes it is possible to combine both techniques within one web application for different parts of data.
Of course you can go for exotic solutions, like using some JavaScript-based server-side framework. In this case you can share the code between the server and the client.
I don't have a complete answer for you; I too have struggled with this on a recent project. But, here is a thought:
Ajax call to Rails
Rails composes the entire grid again, with the new row.
Rails serializes HTML, which is returned to the browser.
Javascript replaces the entire grid element with the new HTML.
Note: I'm not a Rails person, so I'm not sure if those bits fit.
Has anyone tried something like the following? There's redundant data now, but all the rendering is done server-side and all the interaction is done client-side.
Server Side:
render_table_initially:
if nojs:
render big_html_table
else:
render empty_table_with_callback_to_load_table
load_table:
render '{ rows: [
{ foo: "a", bar: "b", renderedHTML: "<tr><td>...</td></tr>" },
{ foo: "c", bar: "d", renderedHTML: "<tr><td>...</td></tr>" },
...
]}'
Client side:
dojo.xhrGet({
url: '/load_table',
handleAs: 'json',
load: function(response) {
dojo.global.all_available_data = response;
dojo.each(response.rows, function(row) {
insert_row(row.renderedHTML);
}
}
});
Storing the all_available_data lets you do sorting, filtering, grouping, etc. without hitting the server.
I'm only cautious because I've never heard of it being done. It seems like there's an anti-pattern lurking in there...
"Perhaps send JSON from the server but also include the HTML snippet as an attribute so the Javascript can do the filtering, sorting, etc. and then just insert the given fragment?"
I recommend keeping the presentation layer on the client and simply downloading data as needed.
For a rudimentary templating engine, you can extend Prototype's Template construct:
http://www.prototypejs.org/api/template
As your client scales and you need a rich and flexible MVC, try PureMVC.
http://puremvc.org/content/view/102/181/
As in regular server side programming you should strive to encapsulate your entities with controls, in your case client side controls that has data properties and also render methods + events.
So for example lets say you have on the page an are that shows tree of employees, effectively you should encapsulate it behavior in a client side class that can get JSON object of the employee list / by default connect to the service and a render method to render the output, events and so on.
In ExtJS its is relatively easy to create these kind of controls - look at this article.
Maybe I'm not fully understanding your problem but this is how I would solve your requirements:
Sort, Filter
It can be done all in JavaScript, without hitting the server. It's a problem of manipulating the table rows, move rows for sorting, hide rows for filtering, there is no need for re-rendering. You will need to mark columns with the data type with custom attributes or extra class names in order to be able to parse numbers or dates.
If your reports are paginated I think it's easier and better to refresh the whole table or page.
Group
I can't help here as I don't understand how will you enable grouping. Are you switching columns to show accumulates? How do you show the data that doesn't support accumulates like text or dates?
New item
This time I would use AJAX with a JSON response to confirm that the new item was correctly saved and possibly with calculated data.
In order to add the new item to the results table I would clone a row and alter the values. Again we don't need to render anything in client-side. The last time I needed something like this I was adding an empty hidden row at the end of the results, this way you will always have a row to clone no matter if the results are empty.
Number 5 in my list of five AJAX styles tends to work pretty well.