Socrata consumerfinance.gov complaintdatabase API JSON parsing - javascript

I want to retrieve consumerfinance.gov complaintdatabase data using the Socrata API.
This is straightforward to retrieve the data using the instructions on the site at http://www.consumerfinance.gov/complaintdatabase/technical-documentation/#api-documentation
I used the following query with '6yuf-367p' to get just 'prepaid card' product data and the '.json' tag to get it in JSON format:
http://data.consumerfinance.gov/api/views/6yuf-367p/rows.json
I used PHP to retrieve data with this query:
$url = "http://data.consumerfinance.gov/api/views/6yuf-367p/rows.json";
$json = file_get_contents($url);
$data = json_decode($json);
var_dump($data);
The results are paraphrased below. A couple of things about the results that are different than what I was expecting.
I wasn't expecting the 'Meta' section. The column names are in the Meta section not associated directly with the data in { key: value } format.
I was expecting the Data section to have { key: value } format instead of just 'values'. This is different than the format described on Socrata help page at
http://dev.socrata.com/docs/formats/json.html
I am not advanced javascript developer so I am wondering how best to proceed. Ideally I want only the 'Data' section with column names in the { key: value } format. I wanted it in that { key: value } format to use with things like Google Charts.
I am imaging I would have to save the column names in array and then loop through each data row and rewrite the data rows with column names included to get { key: value } format.
My other option would be to use the csv API format which is super clean with nice clean columns and no Meta section. But then I would have to convert the csv to JSON which seems unnecessary as the JSON is available.
CSV query is this:
http://data.consumerfinance.gov/api/views/6yuf-367p/rows.csv
So couple of questions:
Does socrata provide an API feed without the 'Meta' section? Is there a filter I can use to exclude the 'Meta' section?
If answer to #1 is no, does Socrata have ready javascript to parse the JSON to get the 'Data' section in { key: value } format similar format as described on Socrata help page?
Thanks!
{
"meta" : {
"view" : {
"id" : "6yuf-367p",
"name" : "Prepaid Card Complaints",
"averageRating" : 0,
"createdAt" : 1434039311,
etc etc
"columns" : [ {
"id" : -1,
"name" : "sid",
"dataTypeName" : "meta_data",
"fieldName" : ":sid",
"position" : 0,
"renderTypeName" : "meta_data",
"format" : {
}
}, {
etc etc
"data" : [ [ 208134, "A7A3941C-A764-44CA-ABC0-66DE814F1969", 208134, 1438091214, "924763", 1438091214, "924763", null, "2015-07-13T00:00:00", "Prepaid card", "General purpose card", "Managing, opening, or closing account", null, null, null, "Amex", "WA", "982XX", "Web", "2015-07-19T00:00:00", "Closed with monetary relief", "Yes", null, "1464043" ]
......
]
}

It looks like you grabbed the wrong JSON URL. The one you grabbed is for the JSON export, which will dump you the entire dataset in a JSON format along with all of the metadata, and it doesn't provide a queryable API endpoint.
Instead, you should use the endpoint https://data.consumerfinance.gov/resource/jhzv-w97w.json. That'll get you the {"key" : "value"} pairs you're looking for, like this:
[ {
"zip_code" : "982XX",
"complaint_id" : "1464043",
"issue" : "Managing, opening, or closing account",
"date_received" : "2015-07-13T00:00:00",
"state" : "WA",
"product" : "Prepaid card",
"company_response" : "Closed with monetary relief",
"company" : "Amex",
"submitted_via" : "Web",
"date_sent_to_company" : "2015-07-19T00:00:00",
"sub_product" : "General purpose card",
"timely" : "Yes"
}, ... ]
You can also view API docs for that dataset at: http://dev.socrata.com/foundry/#/data.consumerfinance.gov/jhzv-w97w
Good luck, and let me know if you have more questions.

Related

MongoDB How to filter db.adminCommand output

does anybody know how to filter mongodb db.adminCommand output? Because if I run this command db.adminCommand({ "currentOp": true, "op" : "query", "planSummary": "COLLSCAN" }) I get a huge JSON output but I'm only interested in some fields ( like secs_running, op, command, $db)
Many thanks!
You can add the filters straight to the command object like the following:
var commandObj = {
"currentOp" : 1,
"waitingForLock" : true,
"$or" : [
{
"op" : {
"$in" : [
"insert",
"update",
"remove"
]
}
},
{
"command.findandmodify" : {
"$exists" : true
}
}
]
};
db.adminCommand(commandObj);
You can see some filter examples on the MongoDB docs: https://docs.mongodb.com/manual/reference/method/db.currentOp/#examples
Just re-read your question and I think you might of meant just projecting fields back from the database that you care about? if that's the case you can just execute a map on top of the current results so you only see what you care about?
db.adminCommand(commandObj).inprog.map(x => x.opid};

Firebase JSON structure not plugging into widget function

I know this is probably a common question but I had issues getting any solutions to work with my use case. I have a calendar widget I have forked on Codepen made using JS. I want to replace the default data with my Firebase database but the JSON structure is different and breaks the Calendar function. My JSON in Firebase is as follows:
{
"Events" : {
"cool event" : {
"calendar" : "Other",
"color" : "yellow",
"date" : "2017-01-13",
"eventName" : "new"
}
},
The original method that the widget received data was as so:
var data = [
{ eventName: 'Lunch Meeting w/ Mark', calendar: 'Work', color: 'orange', date: '2014-02-08' },]
How can I format either my data or modify the function to accept my data structure? Keep in mind, Firebase does not allow a simple array and stores them as objects...
Here is a CodePen: http://codepen.io/Auzy/pen/OWJxqO
One way would be to transform your data to an array of the required format
For example
var firebase = {
"Events" : {
"cool event" : {
"calendar" : "Other",
"color" : "yellow",
"date" : "2017-01-13",
"eventName" : "new"
}
}
};
var data = [];
for(event in firebase.Events){
data.push(firebase.Events[event])
}
console.log(data)
Update
Firebase api is asynchronous, so you need to wait for the data (using the .once('value').then(..)) to return and then transform the data (which you actually do not need since firebase provides an iterator) and then initialize the calendar.
So you need to change your code to
var stuff = firebase.database().ref().child("Events");
var data = [];
stuff.once('value').then(function(snapshot){
snapshot.forEach(function(event){
data.push( event.val() );
})
var calendar = new Calendar('#calendar', data);
});
(and remove the initialization from the bottom of the code as it has been inserted in the callback)

How do I update an array from MongoDB document that match a certain value?

I'm a new mongoDB user, and I'm confused on how to update my document.
I have this document:
"_id" : ObjectId("5674c86aba97df0800995da7"),
"role" : "tutor",
"schools" : [
"CHIJ Our Lady Of Good Counsel",
"Nanyang Technological University"]
And I'd like to update the schools array from this document that matches a certain value from a var. I've tried like this:
var schools = [{ "name": "Nanyang Technological University (NTU)", "value": "nanyang-technological-university-(ntu)" }];
db.getCollection('clients').update(
// query
{
"schools" : schools.name
},
// update
{
$set:{"schools":[schools.value]}
},
// options
{
"multi" : true, // update only one document
"upsert" : false // insert a new document, if no existing document match the query
}
);
But of course, it's not working. Can anyone help me with this? Thanks!
I need the document to be displaying this value after update:
"_id" : ObjectId("5674c86aba97df0800995da7"),
"role" : "tutor",
"schools" : [
"CHIJ Our Lady Of Good Counsel",
"nanyang-technological-university-(ntu)"]
You could make use of the $ operator:
for each item in the schools variable,
find records, with the corresponding name in the schools array field.
$set the new value at the position of the matched name using the $ positional operator.
code:
var schools=[{"name":"Nanyang Technological University",
"value":"nanyang-technological-university-(ntu)"}];
schools.forEach(function(i){
db.clients.update({"schools.name":i.name},
{$set:{"schools.$":i.value}},
{"multi":true})
})

Why this javascript is not returning the restaurant name?

Address query
https://maps.googleapis.com/maps/api/place/textsearch/json?query=restaurant+in+kolkata&key=API_KEY
RESTAURANT JSON LIST
{
"html_attributions" : [
"Listings by \u003ca href=\"http://www.indiacom.com/\"\u003eIndiacom Yellow Pages\u003c/a\u003e"
],
"next_page_token" : "CuQB2wAAALbNaxylA2QNB4pNRsjUxIOYwTSnii1OIwvXqiCPM3736kQZGJ4sEKthpVgvUkskg0ebaLbeE1iNbNKnO7N__X_FpsGprU3o4scQ5aZuUcroSkZVhuWOKvxWHA9IYVXhFmqrVsgG9mjinq-RvANuV6oKzcvnXK09GvdZR0Xp-HINbfoakOVR0TsoOFNCw4UIWrihFSXPJOXeNtTYuImUrPkYKZVt-Y8xMKxr_aOvRR7L0PfcAcXPpSBB2IugIh2K3ESUGMypJD8EuPW1rqqvvYXcMqqHp8iWzq9h1-ytSFl8EhDn2tzr7gU7AkcPORFyLXuiGhSG1bFqUIK2yQb6_fhN-nbym8c17Q",
"results" : [
{
"formatted_address" : "No. 26,Next To Museum,Sudder Street, New Market Area, Jawaharlal Nehru Road, Kolkata, West Bengal 700016, India",
"geometry" : {
"location" : {
"lat" : 22.558687,
"lng" : 88.350889
}
},
"icon" : "http://maps.gstatic.com/mapfiles/place_api/icons/restaurant-71.png",
"id" : "68f2d647334cdaa6a4063c86347252f5a6ebe4c1",
"name" : "Zaranj",
"opening_hours" : {
"open_now" : false,
"weekday_text" : []
},
"place_id" : "ChIJ24Jtvqd3AjoRFb_Bd8Il8Cw",
"rating" : 4.2,
"reference" : "CmRZAAAA9-XMiEF-hYPFZh9kaOBwXQ6R-60XStrKsOEkThALaaeHHmJv6kvEd2JWCh7F9XU9XwyNaNRj6KZQkgGEvhhD8qTIs2fiH-Cng5kmlbIpuQLzQGFgQ8dgnvmL2U1-Uz3REhBb9bJSg-t8S8GDpGUGqlsSGhQnqUVV75IG_AFtVpeV4Xpg2YeTmA",
"types" : [ "restaurant", "food", "establishment" ]
},
JS
//Get Restaurant list
function getRestaurants()
{
var search=$("#search").val();
var prestring="restaurant+in+";
var jsonRList="https://maps.googleapis.com/maps/api/place/textsearch/json?query="+prestring+search+"&key=API_KEY";
alert(jsonRList);//returning the address correctly
$.getJSON(jsonRList, function (data1) {
var Rname=data1.results[0].name;
alert(Rname);//returning nothing
});
}
Why this javascript is not returning the restaurant name?(testing first one results[0])
What is the wrong in it?
alert(Rname);//returning nothing
I think you can use console.log(), alert() or document.write() as long as the information is deserialized, I am using JSON.parse() for deserializing, eval() is what people used before. Deserializing means to recover the data encoded in json so Javascript can use it as an object. In fact, if you use write.document(someJson) you will see [object object] meaning you have all your data there ready to be used, but you need to JSON.parse() it first. JSON.stringnify() is the opposite of what you want to do, because that function is for serialize the data and then transmit it, then the receptor needs to use JSON.parse() to deserialized it and use it in Javascript.
I like to use write.document() in order to see the information, but console log allows you to see what is inside of the object easier. Here, some code example:
// apiData would be the variable with the API information
var externalData = JSON.parse(apiData);
// will output [object object]
document.write(externalData);
// will output “ “
document.write(' ');
// will output the latitude
document.write(externalData.results[0].geometry.location.lat);
// will output in the console, you need to type externalData in the console log
console.log(externalData);
I hope this helps.
Just for clarification I used this answer to respond a similar question in this other forum How to get geocode latitude and longitude from json google map api in variables by javascript?)

How to get data from json in order to save them in database?

I am beginner in Javascript/jQuery and I am working on an interface made with KnockoutJS, so I have several models. I would like to save all the data in the database but I don't know how to do it.
I started with :
self.save = function() {
var data = ko.toJS(self);
var test = ko.toJSON(self);
console.log(test);
}
$.ajax({
url: "myURL",
data: {'carrier': data.carrier},
type: "POST",
});
and this is the result of the console.log :
{"id":1,"carrier":"1","Settings":[{"id":1,"price":{"id":1,"DeliveryStandard":"3.00","DeliveryExpress":"6.00","Details":{"id":1,"Standard":[{"id":1,"fromPrice":0,"maxPrice":"45.000"}],"Express"[{"id":1,"fromPrice":0,"maxPrice":"66.000"}]}}}}]}
I can get the value of carrier by using data.carrier but I don't know how to get the other data like DeiveryStandard, DeliveryExpress, fromPrice, maxPrice ...
Have you got an idea?
Thanks you in advance, and sorry if my question is silly!
If you format your JSON into a more readable format, with indenting, it makes it a lot easier to understand:
(though it should be noted that it is only technically JSON while in a string format, outside of that it is just a standard javascript object)
{
"id":1,
"carrier":"1",
"Settings":[
{
"id":1,
"price": { "id":1,
"DeliveryStandard":"3.00",
"DeliveryExpress":"6.00",
"Details": { "id":1,
"Standard": [{"id":1,
"fromPrice":0,
"maxPrice":"45.000"
}],
"Express" //Missing semi-colon
[{"id":1,
"fromPrice":0,
"maxPrice":"66.000"
}]
}
}
}}//One too many closing braces
]
}
First thing to note is you have 2 syntax errors, highlighted above with comments. So fix them first! (Though I wonder if they are typos as you seem to have it working at your end)
Then we can look at the structure tree to work out where the values you want are...
DeiveryStandard and DeliveryExpress are both properties of an object assigned to price, which it a property of the first item in the Settings array. So you can access them like so:
var DeliveryStandard = data.Settings[0].price.DeliveryStandard;
var DeliveryExpress= data.Settings[0].price.DeliveryExpress;
fromPrice and maxPrice are found multiple times, in both Standard and Express items. So you need to decide what version you need. If you want Standard then you can get the first item of the Standard array like so:
var standardObject = data.Settings[0].price.Details.Standard[0];
Which you can then access the properties of like:
var fromPrice = standardObject.fromPrice;
var maxPrice = standardObject.maxPrice;
I am sure you can work out how to get the Express version of the same data!
From what you seem to have been able to work out on your own, I think your problem is not knowing how to deal with the arrays. Note that arrays are defined with square brackets [], and elements within an array should be accessed with a zero-based index, for example: array[0] for the first element, and array[1] for the second element.
This should work.
console.log(data.Settings[0].price.DeliveryStandard);
Fixed some errors in your JSON.
var j = {
"id" : 1,
"carrier" : "1",
"Settings" : [{
"id" : 1,
"price" : {
"id" : 1,
"DeliveryStandard" : "3.00",
"DeliveryExpress" : "6.00",
"Details" : {
"id" : 1,
"Standard" : [
{
"id" : 1,
"fromPrice" : 0,
"maxPrice" : "45.000"
}
],
"Express": [
{
"id" : 1,
"fromPrice" : 0,
"maxPrice" : "66.000"
}
]
}
}
}
]
};
alert(j.Settings[0].price.DeliveryStandard);
alert(j.Settings[0].price.DeliveryExpress);
alert(j.Settings[0].price.Details.Standard[0].fromPrice);
alert(j.Settings[0].price.Details.Standard[0].maxPrice);

Categories

Resources