Develop Reference

What is the best way to manipulate an API AJAX JSON response and pass it to another page?

I think I have a tough one for you guys. Or at least it's been tough for me. I've been searching for the best way to do this in Stack Overflow and everyone that has asked has been given a different response.
I have this code that is accessing an API and calling a maintenance list of all the vehicles in a fleet.
function getMaintenanceList() {
var settings = {
"url": "API URL HERE",
"method": "GET",
"timeout": 0,
"headers": {
"Authorization": "Bearer token here"
},
};
$.ajax(settings).done(function (response) {
// The response the API sends is a JSON object.
// It is an array.
var jsonMaintenance = response;
var parsedJson = JSON.stringify(jsonMaintenance);
//Left over code from when I was trying to
//pass the data directly into the other page
// I was unable to do so
//return jsonMaintenance;
//Left over code from when this was in a PHP file
//and I was posting the stringified response to the page
// for testing purpose
//I had to disable CORS in Google Chrome to test the response out
//console.log(jsonMaintenance);
//document.getElementById("main").innerHTML = parsedJson;
});
};
The code above works well. What I was attempting to do here was write the stringified response to a file. Save that file in the server. Call it from another page using JavaScript and save it as an object in JavaScript, parse it using JSON.parse(), and then pull the required information.
Here's an explanation as to why I'm trying to do it this way. When I call the maintenance list from the API, I'm getting the entire maintenance list from the API, but I need to be able to display only parts of the information from the list.
On one page, we'll call it vehicle-list.php, on it I have a list of all the vehicles in our fleet. They all have unit numbers assigned to them. When I click on a unit number on this page it'll take me to another page which has more information on the vehicle such as the VIN number, license plate, etc. we'll call this page vehicle-info.php. We're using this page for all the vehicles' information, in other words, when we click on different unit numbers on vehicle-list.php it'll always take us to vehicle-info.php. We're only updating the DOM when we go to the page.
I only want to include the information specific to each vehicle unit in the page along with the other info in the DOM. And I only want to call the info from the API once as I am limited to a certain amount of calls for that API. This is why I am attempting to do it this way.
I will say that what I originally wanted to do was get this JSON response once every 24 hours by using a function in vehicle-list.php save the reponse as a variable as seen above var jsonMaintenance = response; and then just access certain parts of the array every time a unit number is clicked. However, I have been unable to access the variable in any other page. I've written many test files attempting to call jsonMaintenance without success so I've been trying to to just save it as a text file to the server and I haven't been able to figure that out either.
After explaining all of the above. My questions are these:
How do I best manipulate this data to accomplish what I want to accomplish? What would be the best standard? Is the above code even the right way to call the data for what I'm trying to do?
There doesn't seem to be a set standard on accomplishing any of this when I search on Stack Overflow. I'd like to be as efficient as possible.
Thank you for your time.

there is a lot of ways how you pass your data through your website after getting it in from an api call, the best approach is to store these information in a database and call it back in which ever way you want, you can do that as far as you are using php, you can store it to sql or to access, if you don't want to store these information in a database like in sql or access, then best way is to store it to localStorage and call it back whenever you want.
I will show you briefly how you can do that, if you want better explanation post an example of your returned data.
to store an item in localstorage use,
localStorage.setItem('key', 'value');
to call an item back from localstorage use,
var somevar = localStorage.getItem('key')
to remove specific item from localstorage use,
localStorage.removeItem('key')
to clear all items saved to localstorage use,
localStorage.clear()
be aware storing the data to localStorage is only at the station you are using

I would do it somehow like this.
Call the maintenance list from the API with the server side language of your choice which seems to be PHP in your case. Lets say the script is called: get-list.php. This can be triggered by a cron job running get-list.php in intervals limited to the certain amount of calls that you are allowed to do for that API. Or if you are not able to create cron jobs then trigger the same get-list.php with an AJAX-call (eg jQuery.get('sld.tld/get-list.php') - in this case get-list.php have to figure out if its the right time to call the API or not).
Now that you have the data you can prepare it as you want and store it as a JSON-string in a text file or database of your choice. If I get you right you have a specific dataset for each vehicle, which have to be identified by an id (you named it "unit number") so your JSON would look kind of: {"unit1": { property1: "val1", property2: "val2" }, "unit2": { property1: "valXYZ", property2: "valABC" }} or alike.
Now when you link to vehicle-info.php from vehicle-list.php, you do it like so: ancor or similar as well. Of course you can also grab the data with AJAX, its just important to deliver vehicle-info.php the corresponding unit number (or id - better to say) and you are good to go.
vehicle-info.php now have all there is to render the page, which is the complete data set stored in text file or data base and the id (unit number) to know which part of the whole dataset to extract.
I wanted to give you this different approach because in my experience this should work out just so much better. If you are working server side (eg PHP) you have write permissions which is not the case for JavaScript-client side. And also performance is not so much of an issue. For instance its not an issue if you have heavy manipulating on the data set at the get-list.php-level. It can run for minutes and once its done it stores the ready-to-use-data making it staticly available without any further impact on performance.
Hope it helps!

If i ran into a similiar problem i would just store the data in a database of my own and call it from there, considering you are only (willing/abe/allowed) to request the data from the API very rarely but need to operate on the data quite frequently (whenever someone clicks on a specific vehice on your applicaiton) this seems like the best course of action.
So rather than querying the data on client side, I'd call it from server, store it on server and and have the client operate on that data.

Old data using YQL and Javascript getJSON call

After doing some searching, it seems that this type of issue affects many users out there so an answer to this question could help many users of Yahoo's YQL Platform.
I am essentially aiming to extract a semi-static CVS document stored on a webserver to then parse in Javascript. Semi-static means that the CVS document isn't getting appended to with additional entries, rather each entry is getting modified.
Using the YQL console https://developer.yahoo.com/yql/console/ gives me updated data with every call made to my semi-static CVS file stored on a remote server. I can modify the data and YQL console will successfully return the updated data. When I extract their provided rest query and simply 'paste' it into a browser window, the data provided correspond to the very first query that I made. When I embed the query as a $.getJSON request in Javascript as such:
$.getJSON("https://query.yahooapis.com/v1/public/yqlq=select%20*%20from%20csv%20where%20url%3D'mywebsite.csv'&format=json").done(function (data) {
I still get the outdated data. If I switch to a different web-browser or device, the information is still outdated which gives me the feeling that it is not a cache issue on the local machine.
I believe the problem is in one of two spots:
1) Perhaps Yahoo caches the queries and only acquires updated information from tables/files that grow dynamically
2) I am not using the YQL query correctly.
As an additional note, the exact same query structure works perfectly with Google Forms (which can export as a CVS) and also works without a hitch on a dynamically growing CVS document that I used on a now antiqued database, requiring a quick switch to the simple semi-static document.
Any thoughts or fixes that can work on my semi-static CVS document?

This sounds like a browser cache issue
Try adding a timestamp in the url which makes every request have a unique url and therefore browser won't have a cache for it
var params ={
q : "select%20*%20from%20csv%20where%20url%3D'mywebsite.csv'",
format: "json",
_v : Date.now()
};
$.getJSON("https://query.yahooapis.com/v1/public/yql", params ).done...

Where to put "a lot" of data, array / file / somewhere else, in JS on node.js

This may be a "stupid" question to ask, but I am working with a "a lot" of data for the first time.
What I want to do: Querying the World Bank API
Problem: The API is very unflexible when it comes to searching/filtering... I could query every country/indicator for it self, but I would generate a lot of calls. So I wanted to download all informations abourt a country or indicator at once and then sort them on the machine.
My Question: Where/How to store the data? Can I simply but it into an array, do I have to worry about size? Should I write to a temporary json file ? Or do you have another idea ?
Thanks for your time!
Example:
20 Countries, 15 Indicators
If I would query every country for itself I would generate 20*15 API calls, if I would call ALL countries for 1 indicator it would result in 15 API calls. I would get a lot of "junk" data :/

You can keep the data in RAM in an appropriate data structure (array or object) if the following are true:
The data is only needed temporarily (during one particular operation) or can easily be retrieved again if your server restarts.
If you have enough available RAM for your node.js process to store the data in RAM. In a typical server environment, there might be more than a GB of RAM available. I wouldn't recommend using all of that, but you could easily use 100MB of that for data storage.
Keeping it in RAM will likely make it faster and easier to interact with than storing it on disk. The data will, obviously, not be persistent across server restarts if it is in RAM.
If the data is needed long term and you only want to fetch it once and then have access to the data over and over again even if your server restarts of if the data is more than hundreds of MBs or if your server environment does not have a lot of RAM, then you will want to write the data to an appropriate database where it will persist and you can query it as needed.
If you don't know how large your data will be, you can write code to temporarily put it in an array/object and observe the memory usage of your node.js process after the data has been loaded.

I would suggest storing it in a nosql database, since you'll be working with JSON, and querying from there.
mongodb is very 'node friendly' - there's the native driver - https://github.com/mongodb/node-mongodb-native
or mongoose

Storing data from an external source you don't control brings with it the complexity of keeping the data in sync if the data happens to change. Without knowing your use case or the API it's hard to make recommendations. For example, are you sure you need the entire data set? Is there a way to filter down the data based on information you already have (user input, etc)?

Architecture for temporary storing of values within a javascript library

I am currently writing a javascript library that wraps a REST API provided by a third party (intended to be used on server side, but wouldn't like to limit it to). One of the actions defined by the api is 'login' which gives me a session key that I need to use on further requests. Currently, everytime I go to use this library I need to login again as there is no persistence of this session key. My question is, what is the best way to persist it throughout a session?
My first instinct was to give the library a callback that would store it and a callback that would retrieve it and the implementation can determine how that session key is persisted:
var thirdPartyApi = new ThirdPartyApi({
loginCredentials: {..},
setSessionKeyCallback: function() {},
getSessionKeyCallback: function() {}
});
thirdPartyApi.makeSomeRequest('foo');
Can you recommend the best architecture for this problem?

It seems like you want to use the REST Api in a browser. There are some factors you need to take into account, such as, navigating away from the page and coming back to it later.
You can use Web Storage to store the key. There are two types, localStorage and sessionStorage. The only difference between then is that sessionStorage is deleted when the browser window is closed, while localStorage isn't. Web Storage is supported by all modern browsers and IE8+ http://www.w3schools.com/html/html5_webstorage.asp
The localStorage object can be used as such:
localStorage.setItem("bar", foo);
var foo = localStorage.getItem("bar");
localStorage.removeItem("bar");
sessionStorage object can be used the same way.
both localStorage and sessionStorage are global objects that can be accessed from anywhere, so there is no need for any special architecture on your ThirdPartyApi object.

Possible to cache JSON to increase performance / load time?

I'm using a JSON file to autopopulate a drop down list. It's by no means massive (3000 lines and growing) but the time taken to refresh the page is becoming very noticeable.
The first time the page is loaded the JSON is read, depending on what option the user has selected dictates which part of the JSON is used to populate the drop down.
It's then loaded on every refresh or menu selection after. Is it possible to somehow cache the values to prevent the need for it to be reloaded time and time again?
Thanks.
EDIT: More Info:
It's essentially a unit converter. The JSON holds all the details. When a users selects 'Temp' for example a call is made and the lists are populated. Once a conversion is complete you can spend all day running temp conversions and they'll be fine but everytime a user changes conversion type so now length, the page refreshes and takes a noticeable amount of time.

Unfortunately, I don't know of a standardized global caching mechanism in PHP. This article says that Optimizer Plus, a third party accelerator, is being included in core PHP starting in version 5.5. Not sure what version you are using but you could try that.
On a different note, have you considered file storage as andrew pointed out? I think it combined with $_SESSION could really help you in this case. Let me give you an example that would work with your existing JSON data:
Server Side
Store your JSON data in a .json file on your PHP server:
{
"data": "some data",
"data2": "more data",
"data3": [
...
],
etc.
}
Note: Make sure to properly format your JSON data. Remember all strings must be enclosed in double quotes ".
In PHP, use an if statement to decide the appropriate action:
error_reporting(E_ALL);
ini_set("display_errors", "On");
session_start();
if(isset($_SESSION['dataCache'])) {
echo json_encode($_SESSION['dataCache']);
} else {
$file = 'data.json';
if (!is_file($file) || !is_readable($file)) {
die("File not accessible.");
}
$contents = file_get_contents($file);
$_SESSION['dataCache'] = json_decode($contents, true);
echo $contents;
}
So lets dig into the above coding a little more. So here's what we are doing in a nutshell:
Turn on error reporting and start session support.
Check to see if we've already read the file for this user.
If so, pull the value from storage and echo it out and exit. If not continue below.
Save off the file name and do a little error checking to ensure PHP can find, open and read the contents of the file.
Read the file contents.
Save the decoded json, which is not an array because of the `true` parameter passed to `json_decode`, into your `$_SESSION` variable.
Echo the contents to the screen.
This will save you the time and hazzle of parsing out JSON data and/or building it manually on the server. It will be cached for the users session so that they can use it through out.
Client Side
I assume you are using ajax to fetch the information? If not correct me, but I was assuming that's where some of your JavaScript comes into play. If so you may consider this:
Store the returned data in sessionStorage on the user's browser when it's returned from the server:
$.ajax({
...
success: function (res) {
localStorage.setItem("dataCache", JSON.stringify(res));
},
...
});
Or if you use promise objects:
$.ajax({
...
}).done(function (res) {
localStorage.setItem("dataCache", JSON.stringify(res));
});
When you need to read it you can do a simple test:
var data;
// This returns null if the item is not in local storage.
// Since JavaScript is truthy falsy, it will be evaluated as false.
if(localStorage.getItem("dataCache")) {
data = JSON.parse(localStorage.getItem("dataCache"));
} else {
// Make ajax call, fetch object and store in localStorage in the success or done callbacks as described above
}
Notes:
localStorage is a new feature in HTML5, so it's not fully supported on all browsers yet. Most of the major ones do however, even as far back as IE8 (I think). However, there is no standardized size limit on how much these browsers are required to hold per site.
It's important to take that into consideration. I can guarantee you probably will not be able to store the entire 30,000 line string in localStorage. However, you could use this as a start. Combined with the server side solution, you should see a performance increase.
Hope this helps.

I use the browser's cache to ensure that my large chunk of JSON is only downloaded once per session. I program in ASP.NET, but I'm sure PHP has the same mechanisms:
On session start, I generate a random string as session key for my dynamic JavaScripts. This key get stored in the ASP.NET session state under the key JsonSessionID. That way I can refer to it in my page markup.
I have a "generic http handler" (an ashx file) that when called by the browser, returns a .js file containing my JSON.
In my HTML I include the dynamic script:
<script type="text/javascript" src="/dynamicJSON.ashx?v=<%= JsonSessionID %>"></script>
The browser will automatically cache any URLs included as scripts. The next time the browser is asked to load a cached script from a URL, it will just load up the file from the local disk. This includes dynamic pages like this.
By adding the ?v= in there, I ensure that the JSON is updated once per session.
Edit
I just realized that your JSON is probably static. If that's the case, you can just put your JSON into a static .js file that you include in your HTML, and the browser will cache it.
// conversionData.js
var conversionData = { "a":1,"b":2,"c":3 };
When you include the conversionData.js, the conversionData variable will be in scope with the rest of your page's JavaScript that dynamically updates the drop-downs.
Edit 2
If you are serving static files, this blog post has a good pattern for cache-busting based on the file's date modified property. i.e. the file is only downloaded when it is changed on the server.
I have yet to find a good method for cache-busting JSON created via database lookup tables, other than per-session. Which isn't ideal because the database could change mid-session.

Once you've got your JSON data decoded into an object you can just keep the object around, it should persist until a page reload at least.
If you want to persist between reloads you might want to look at HTML5's localStorage etc.
You would need to come up with an age strategy, maybe just dump the current date in there with it as well so you can compare that and expire as needed.

I would suggest storing your json data to a session. On first page load you can write a script to get your json data, then store them into a session.
on each page load/refresh afterwards you can check our session to decide what to do - use the session data or fetch again your json data.
This approach suites me for small scale data (for example: an array of products - colors - sizes - prices).
Based on your data you should test you loading times.

Here is a simple hack:
Create a call to a php file as GET request with parameter "bla-bla.html"
or "bla-bla.css"... well you know, it makes browser think it is not a php, but rather "html" or "css". And browser will cache it.
To verify that the trick is working - go to the "network" tab of the browser dev panel and you will see column "type" there along with "transferred" - instead of having php there and actual size, you will find "html" and "(cached)"
This is also good to know when you passing parameters like "blah-blak.html" to the php file and expect it will not be cached. Well, it will be cached.
Tested on FireFox Quantum 57.0.1 (Mac 64bit)
P.S.
Chrome 63 on Mac is capable of recognising real file type in this situation. So it cannot be fooled.

Thinking out of the box here:
but if your list has 3000 lines and growing (as you said)
is it possible for you to establish its maximum size ?
let's say the answer is 10,000 (max) items; then do you really need an ajax call ?
you could transfer the data straight away with the page
(depending on your architecture of course, you could come out with different solution)