We use the D3 JavaScript to initialize data documents, then build an application-specific JavaScript to process data.
A subset of the application-specific JavaScript looks like this:
drawLegend();
thousand_sep_format = d3.format(',');
d3.json("http://wafi.iit.cnr.it/webvis/tmp/dbpedia/realOntology.json", function(error, root)
More specifically it correctly processes this JSON file:
http://wafi.iit.cnr.it/webvis/tmp/dbpedia/realOntology.json
However, when we copy the original JSON file to another Linux/Ubuntu server, the copied JSON file cannot be processed.
Here is the copied JSON file:
http://www.ontomatica.com/public/test/dbpedia_ontology/realOntology.json
What is the difference between JSON-original and JSON-copied?
What is the correct process to copy an original JSON file to a new server?
Our goal is to remove sections of the original JSON file and then plot the subset. Therefore we have to put a working subset on a server.
The first site responds with an Access-Control-Allow-Origin header with the value *. That tells browsers that they should allow xhr access to the site regardless of the originating domain.
The other site doesn't do that, so the browser won't fetch the content.
The problem has nothing to do with the URLs as such, nor with the JSON content. It's a matter of server configuration, and exactly how you change that depends on the hosting environment.
Related
I have imported a JS file that purely is used to contain data. I want to be able to push changes to the data file, is this possible client / browser side only or do I need to use PHP / Python / Node?
<script src="datafile.js"></script>
Data File Contents:
var data = {"someValue":{"someOtherValue": "yeah"},
"anotherValue": [{"key": "value"},{"key": "value"}]}
So in brief to answer my own question based off the answers here:
Editing a file via a client side language is impossible and would
pose a security threat to the server.
If I would like to update the data in the file I will need to use a server side language as I have mentioned. PHP is what I will
be using in my case. My html file will just become a .php and will
listen for $_GET['idToChangeInJSONFile'].
Another option is that I could store the data in localStorage and be able to manipulate from there. This however is not consistent
and would get cleared when a different browser is used or the
browser is reset.
I'm trying to figure out the best way to accomplish the following:
Download a large XML (1GB) file on daily basis from a third-party website
Convert that XML file to relational database on my server
Add functionality to search the database
For the first part, is this something that would need to be done manually, or could it be accomplished with a cron?
Most of the questions and answers related to XML and relational databases refer to Python or PHP. Could this be done with javascript/nodejs as well?
If this question is better suited for a different StackExchange forum, please let me know and I will move it there instead.
Below is a sample of the xml code:
<case-file>
<serial-number>123456789</serial-number>
<transaction-date>20150101</transaction-date>
<case-file-header>
<filing-date>20140101</filing-date>
</case-file-header>
<case-file-statements>
<case-file-statement>
<code>AQ123</code>
<text>Case file statement text</text>
</case-file-statement>
<case-file-statement>
<code>BC345</code>
<text>Case file statement text</text>
</case-file-statement>
</case-file-statements>
<classifications>
<classification>
<international-code-total-no>1</international-code-total-no>
<primary-code>025</primary-code>
</classification>
</classifications>
</case-file>
Here's some more information about how these files will be used:
All XML files will be in the same format. There are probably a few dozen elements within each record. The files are updated by a third party on a daily basis (and are available as zipped files on the third-party website). Each day's file represents new case files as well as updated case files.
The goal is to allow a user to search for information and organize those search results on the page (or in a generated pdf/excel file). For example, a user might want to see all case files that include a particular word within the <text> element. Or a user might want to see all case files that include primary code 025 (<primary-code> element) and that were filed after a particular date (<filing-date> element).
The only data entered into the database will be from the XML files--users won't be adding any of their own information to the database.
All steps could certainly be accomplished using node.js. There are modules available that will help you with each of these tasks:
node-cron: lets you easily set up cron tasks in your node program. Another option would be to set up a cron task on your operating system (lots of resources available for your favourite OS).
download: module to easily download files from a URL.
xml-stream: allows you to stream a file and register events that fire when the parser encounters certain XML elements. I have successfully used this module to parse KML files (granted they were significantly smaller than your files).
node-postgres: node client for PostgreSQL (I am sure there are clients for many other common RDBMS, PG is the only one I have used so far).
Most of these modules have pretty great examples that will get you started. Here's how you would probably set up the XML streaming part:
var XmlStream = require('xml-stream');
var xml = fs.createReadStream('path/to/file/on/disk'); // or stream directly from your online source
var xmlStream = new XmlStream(xml);
xmlStream.on('endElement case-file', function(element) {
// create and execute SQL query/queries here for this element
});
xmlStream.on('end', function() {
// done reading elements
// do further processing / query database, etc.
});
Are you sure you need to put the data in a relational database, or do you just want to search it in general?
There don't seem to be any actual relations in the data, so it might be simpler to put it in a document search index such as ElasticSearch.
Any automatic XML to JSON converter would probably produce suitable output. The large file size is an issue. This library, despite its summary saying "not streaming", is actually streaming if you inspect the source code, so it would work for you.
I had task with xml files as you wrote. This are principals I used:
All incoming files I stored as is in DB (XMLTYPE), because I need a source file info;
All incoming files parsed with XSL transformation. For example, I see that it is three entity here: fileInfo, fileCases, fileClassification. You can write XSL transformation to compile source file info in 3 entity types (in tags FileInfo, FileCases, FileClassification);
When you have output transformed XML you can make 3 procedures, that inserts data into DB (each entity in DB area).
I have a Python program that generates an html page for reporting results. The html page is saved in an output directory on disk alongside a javascript file that helps with dynamic table handling. I also save a JSON file to this output directory that I would like to read in with my javascript file. This JSON file has data from the Python run (saved dictionary) that I would like to be able to access. So in an output directory on disk I have:
C:/somedirectory/output/report.html
C:/somedirectory/output/tables.js
C:/somedirectory/output/data.json
All files have been created from my program.
My html page has a table with checkboxes and if those checkboxes are selected I would like to update a second table based on data saved in the JSON file. Thus I would like to open my html report in any browser and read in the JSON file as a javascript object.
I have been trying to use ajax and .getJSON but am getting the
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access.
I have searched and seen many similar problems but have not come across anything that quite fits what I need. Thoughts and a work around would be greatly appreciated. Thanks.
Update
Since everything is run locally on the client side I have decided to embed the JSON data (python dictionary) and javascript code directly into the html report output. This way the data is internally accessible and the html file can be passed around without dependency issues. The user with the answer I selected below has a link that eludes to this solution.
JavaScript runs on the client machine, hence it can only access files on the client machine using a special setup.
If you want it to read JSON on your server, you should use the path:
http://example.com/output/data.json
Better way would be to read/write JSON file from Python and then send the table data to JavaScript as in this answer: Send data from Python to Javascript (JSON)
I'm just beginning with JavaScript and I was wondering if I can access a JSON file on the localhost without the use of any servers (like WAMP). I was planning to just read the contents of a JSON file and reflect its contents to an HTML file.
Short answer, yes you can.
Make an XMLHttpRequest against a static file such as: resources/myJSON.json (the extension is merely for organization, you can call the file whatever you want really).
Parse the response as JSON.
Obviously the file must contain a properly formatted JSON object of data, but as long as that's the case, you can load static JSON from a file to "simulate" remote connectivity to a server.
When you're browsing a file as: file:// on your system, it should treat the relative paths correctly.
You can grab data from here. Instead of saving JSON as a separate variable within the script file, is it possible to create a file pointer, and populate a Tree with one config option? Saving JSON in a variable does the job, but it's quite cumbersome!
You can't use files outside of things like AIR it's a security risk and the sandbox won't allow it. You can of course keep the JSON in an external file and request it through HTTP using a TreeLoader with AsyncTreeNode and preloading children OR simply evaluate the AJAX response yourself.