Develop Reference

Which encoding does application/dns-message use?

I am writing DNS-over-HTTPS server which should resolve custom names, not just proxy them to some other DoH server, like Google's. I am having trouble properly decoding the body of the request.
For example, I get body of request, that is in binary format, specifically in javascript in Uint8 ArrayBuffer type. I am using the following code to get base64 format of the array:
function _arrayBufferToBase64(buffer) {
var binary = '';
var bytes = new Uint8Array(buffer);
var len = bytes.byteLength;
for (var i = 0; i < len; i++) {
binary += String.fromCharCode(bytes[i]);
}
return btoa(binary);
}
And I get something like this as a result:
AAABAAABAAAAAAABCmFwbngtbWF0Y2gGZG90b21pA2NvbQAAAQABAAApEAAAAAAAAE4ADABKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
Now, per RCF8484 standard this should be decoded as base64url, but when I decode it as such, I get the following:
apnx-matchdotomicom)NJ
I also used this "tutorial" as the reference, but they also decode similarly formatted blob and I get similar nonsense like previously.
There is very little to no information about something like this on the internet and if it is of any help DoH standard uses application/dns-message media type for the body.
If anyone has some insight on what I am doing wrong or how I could edit the question to make it more clear, please help me, cheers :)

As stated in the RFC:
Definition of the "application/dns-message" Media Type
The data payload for the "application/dns-message" media type is a
single message of the DNS on-the-wire format defined in Section 4.2.1
of [RFC1035], which in turn refers to the full wire format defined in
Section 4.1 of that RFC.
So what you get is exactly what is sent on the wire in the normal DNS over 53 case.
I would recommend you use a DNS library that should have a from_wire or similar method to which you can feed this content and get back some structured data.
Showing an example in Python with the content you gave:
In [1]: import base64
In [3]: import dns.message
In [5]: payload = 'AAABAAABAAAAAAABCmFwbngtbWF0Y2gGZG90b21pA2NvbQAAAQABAAApEAAAAAAAAE4ADABKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
In [7]: raw = base64.b64decode(payload)
In [9]: msg = dns.message.from_wire(raw)
In [10]: print msg
id 0
opcode QUERY
rcode NOERROR
flags RD
edns 0
payload 4096
option Generic 12
;QUESTION
apnx-match.dotomi.com. IN A
;ANSWER
;AUTHORITY
;ADDITIONAL
So your message is a DNS query for the A record type on name apnx-match.dotomi.com.
Also about:
I am writing DNS-over-HTTPS server which should resolve custom names,
If you don't do that to learn (which is a fine goal), note that there are already various open source nameservers software that do DOH so you don't need to reinvent it. For example: https://blog.nlnetlabs.nl/dns-over-https-in-unbound/

Converting PHP object to JSON object using only Javascript

I am making a mobile app with Phonegap and using Wordpress as a backend. I am using Advanced Custom Fields with a Google Maps post field which returns a PHP object to the app using JSON API. My Wordpress backend sends a normal JSON object to the app, but inside that object is where a stringified PHP object is returned.
I need to convert the PHP object to a JSON object somehow on the client side(the app which is not in Wordpress). I have looked at other answers that say to use json_encode for this but my problem is that the app is just HTML/Javascript and no PHP. Is there a way to use PHP code in the middle of a Javascript function to do this? Or would it be better to change the backend so that it returns a JSON object instead of a PHP object in the first place? If so, how do I do that?
My experience in PHP is still somewhat limited so any help is appreciated.
edit: To clarify a bit more, I am using Wordpress on a separate domain from my Phonegap app and only using the JSON API plugin on the Wordpress end. I am then using jQuery Ajax calls to retrieve data from the Wordpress backend.
Also the returned PHP object looks like this: a:3:{s:7:\"address\";s:48:\"8915 W 159th St, Orland Hills, IL, United States\";s:3:\"lat\";s:17:\"41.60111599999999\";s:3:\"lng\";s:11:\"-87.8364575\";}
Another way I just thought of as well, would it be possible to just leave it as a PHP object and still read out the values from it somehow? I don't NEED it to be a JSON array, I just need a way to read the individual elements in the array in one way or another.
Here is also a tiny snippet of the JSON returned to clarify what I'm talking about.
"custom_fields": {
"location": [
"a:3:{s:7:\"address\";s:48:\"8915 W 159th St, Orland Hills, IL, United States\";s:3:\"lat\";s:17:\"41.60111599999999\";s:3:\"lng\";s:11:\"-87.8364575\";}"
]
}
That of course isn't the entire JSON object but it gives you an idea of what I'm dealing with.

I know you have a solution that works on the front end, but I still think it'd be better to fix this on the server.
Based on our conversation in the comments, I've had a closer look the code in the WordPress forum. The problem seems to be that the location field is an array of strings, not just a string. maybe_unserialize (and is_serialized, which it uses) don't handle arrays. Here's the updated code, which you should be able to drop into your theme's functions.php. I did a quick test, and it works for me.
class unserialize_php_arrays_before_sending_json {
function __construct() {
add_action( 'json_api_import_wp_post',
array( $this, 'json_api_import_wp_post' ),
10,
2 );
}
function json_api_import_wp_post( $JSON_API_Post, $wp_post ) {
foreach ( $JSON_API_Post->custom_fields as $key => $custom_field ) {
if (is_array($custom_field)) {
$unserialized_array = array();
foreach($custom_field as $field_key => $field_value) {
$unserialized_array[$field_key] = maybe_unserialize( $field_value );
}
$JSON_API_Post->custom_fields->$key = $unserialized_array;
}
else {
$JSON_API_Post->custom_fields->$key = maybe_unserialize( $custom_field );
}
}
}
}
new unserialize_php_arrays_before_sending_json();

If you're using a JSON API to retrieve the data, then why don't you deliver the data in JSON format to your app? Otherwise you seem to remove much of the point of using an API in the first place... You could of course parse that string in JavaScript if you really want to but that's a very ugly and error prone solution.
The JSON API plugin does seem to use JSON:
https://wordpress.org/plugins/json-api/screenshots/

I need to convert the PHP object to a JSON object somehow on the client side(the app which is not in Wordpress).
This bit here leaves me confused. You do not have PHP objects on the client-side, PHP is a back-end technology. What is returned to the client is a string which can be HTML, XML, JSON, plaintext on any other form of encoding.
That said, saying you have an object $obj in PHP, you could pass it to your front-end application creating an end-point retrieve_object.php and in there:
echo json_encode($obj);
So long as that is the only thing your are outputting, you lient-side app can make a request (Eg: AJAX) to retrieve_object.php and get the json object.
BUT , and this is important (!) in doing so you serialize object properties. You will lose any PHP object method. If any object property is an object itself (EG: A DB Connection) then this will be lost too.
Hope this helps!

Transforming JSON in a node stream with a map or template

I'm relatively new to Javascript and Node and I like to learn by doing, but my lack of awareness of Javascript design patterns makes me wary of trying to reinvent the wheel, I'd like to know from the community if what I want to do is already present in some form or another, I'm not looking for specific code for the example below, just a nudge in the right direction and what I should be searching for.
I basically want to create my own private IFTTT/Zapier for plugging data from one API to another.
I'm using the node module request to GET data from one API and then POST to another.
request supports streaming to do neat things like this:
request.get('http://example.com/api')
.pipe(request.put('http://example.com/api2'));
In between those two requests, I'd like to pipe the JSON through a transform, cherry picking the key/value pairs that I need and changing the keys to what the destination API is expecting.
request.get('http://example.com/api')
.pipe(apiToApi2Map)
.pipe(request.put('http://example.com/api2'));
Here's a JSON sample from the source API: http://pastebin.com/iKYTJCYk
And this is what I'd like to send forward: http://pastebin.com/133RhSJT
The transformed JSON in this case takes the keys from the value of each objects "attribute" key and the value from each objects "value" key.
So my questions:
Is there a framework, library or module that will make the transform step easier?
Is streaming the way I should be approaching this? It seems like an elegant way to do it, as I've created some Javascript wrapper functions with request to easily access API methods, I just need to figure out the middle step.
Would it be possible to create "templates" or "maps" for these transforms? Say I want to change the source or destination API, it would be nice to create a new file that maps the source to destination key/values required.
Hope the community can help and I'm open to any and all suggestions! :)
This is an Open Source project I'm working on, so if anyone would like to get involved, just get in touch.

Yes you're definitely on the right track. There are two stream libs I would point you towards, through which makes it easier to define your own streams, and JSONStream which helps to convert a binary stream (like what you get from request.get) into a stream of parsed JSON documents. Here's an example using both of those to get you started:
var through = require('through');
var request = require('request');
var JSONStream = require('JSONStream');
var _ = require('underscore');
// Our function(doc) here will get called to handle each
// incoming document int he attributes array of the JSON stream
var transformer = through(function(doc) {
var steps = _.findWhere(doc.items, {
label: "Steps"
});
var activeMinutes = _.findWhere(doc.items, {
label: "Active minutes"
});
var stepsGoal = _.findWhere(doc.items, {
label: "Steps goal"
});
// Push the transformed document into the outgoing stream
this.queue({
steps: steps.value,
activeMinutes: activeMinutes.value,
stepsGoal: stepsGoal.value
});
});
request
.get('http://example.com/api')
// The attributes.* here will split the JSON stream into chunks
// where each chunk is an element of the array
.pipe(JSONStream.parse('attributes.*'))
.pipe(transformer)
.pipe(request.put('http://example.com/api2'));

As Andrew pointed out there's through or event-stream, however I made something even easier to use, scramjet. It works the same way as through, but it's API is nearly identical to Arrays, so you can use map and filter methods easily.
The code for your example would be:
DataStream
.pipeline(
request.get('http://example.com/api'),
JSONStream.parse('attributes.items.*')
)
.filter((item) => item.attibute) // filter out ones without attribute
.reduce((acc, item) => {
acc[item.attribute] = item.value;
return acc;
.then((result) => request.put('http://example.com/api2', result))
;
I guess this is a little easier to use - however in this example you do accumulate the data into an object - so if the JSON's are actually much longer than this, you may want to turn it back into a JSONStream again.

How can I add some simple dynamic elements to an otherwise static page?

I'm revamping a site that allows booking of events that run each year. Each event has its own page which is currently entirely static: each page consists of a heading, description, and a list of dates sorted by venue. Each year when new dates become available, someone has to go in and manually change the HTML for each one, an obviously laborious task.
I'd like to automate the process somewhat by having, say, a CSV file that stores the dates (which could be added to piecemeal), and then the page grabs the relevant dates from there when it loads. I have no experience with server-side stuff, but I have a little knowledge of jQuery, and I have a feeling I should be able to do this with AJAX or the like - but how?

I think that ivoszz's idea is the best in your case. If you want to get a database and PHP, you will need a way to get your data into the database itself, which opens a whole new can of worms. Sure, database + server-side frontend is the industry standard, but I feel that it is oversized for your requirements.
It is easier to learn how to display JSON with jQuery, when reading it from a simple text file. You only need to write this code once.
Then, whenever there is a change, you can use a simple workflow: You use Excel to enter the events, using a prerecorded format. Then you export the Excel file as .csv. Use a small program to read the CSV and serialize it to JSON. Copy the ouput to a predetermined location on the server. Everything is ready.
Should somebody else have to update the site in your absence, all they need is Excel, the conversion tool (which is tiny), and the server password. I am posting the code for the conversion tool at the end of this answer.
Alternatively, you can use the code to create an ASP .NET WebForms project. Instead of serializing the objects created by the code, you can make an .aspx page and display the data on it using server-side code. However, this has some disadvantages.
.net webforms has a steeper learning curve than JavaScript, and the resulting code will probably use server-side controls, which are hard to style with CSS unless you know how to do it right.
if you are using an inexpensive hosting package instead of your own server, you have to make sure that the provider has the needed .net version on their server. Also, it will probably eat up more space because of all the libraries normally included in .net web projects.
if the structure of the input data never changes, you can compile the converter once and start treating it as a black box. Any changes to the way stuff is displayed can be made in the JSON-reading code and debugged directly in your browser. For a .net solution, you have to keep an installation of Visual Studio around.
.net webforms is not a future-proof skill. Microsoft has created a new, more convenient Web technology, .NET MVC, and I wouldn't suggest starting learning the older tech now. On the other hand, it is not a good idea to make this project MVC if you already have a bunch of existing static pages, because MVC cannot mix static and dynamic pages easily. You'll probably be able to use the content, but will have to rewrite the whole routing system and replace every single internal link.
Here is the code for a C# appliaction which will convert the csv to JSON. When compiled, place the .exe in the same directory as your csv, called DataSource.csv. Doubleclick it. It will produce a new file called autoOutput.json in the same directory. Each line in the .csv file has to be built in the event name; venue; date; cost; format. You can add comments or similar in the Excel to the right of cost, they will be discarded. The order of the lines does not matter. As long as an event name is unique, all venues and dates which start with it will be interpreted as belonging to that event. As long as the combination of event name and venue is unique, all dates will be interpreted as being about that event at that venue.
I'm not doing anything with the information about which lines could not be read because they were too short. You can append it to the file, or exchange its contents with the warning. Then the person who does the conversion will have to fiddle with the csv until there are no warnings, or you can try to leave it in the file, but ignore it when loading for dispalying.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Web.Script.Serialization;
namespace ElendilEvents2JSON
{
public class Event
{
public String Name { get; set; }
public List<EventInVenue> venues { get; set; }
}
public class EventInVenue
{
public String VenueName { get; set; }
public List<EventInstance> Dates { get; set; }
}
public class EventInstance
{
public String When { get; set; }
public String Cost { get; set; }
}
class Program
{
static void Main(String[] args)
{
//read the file
List<int> unreadable;
List<Event> events = readFile(#".\SourceData.csv", out unreadable);
//write the file using the normal JSON serializer. Will output just everything as a single line. If the data structure is changed, it will output in the new structure.
string autoOutput;
JavaScriptSerializer serializer = new JavaScriptSerializer();
autoOutput = serializer.Serialize(events);
File.WriteAllText(#".\autoOutput.json", autoOutput);
}
public static List<Event> readFile(string path, out List<int> unreadableLines)
{
//get the contents out of the file
var lines = System.IO.File.ReadLines(path);
// split each line into an array of strings
var csv = lines
.Select(line => line.Split(';'))
.ToArray();
//will hold all events
List<Event> events = new List<Event>();
//will hold the numbers of all lines which were OK
List<int> unreadable = new List<int>();
//read each line, if you want to skip header lines, change the zero
for (int lineCounter = 0; lineCounter < csv.Length; lineCounter++)
{
string[] line = csv[lineCounter];
if (line.Length >= 4)
{
string eventName = line[0];
Event currentEvent;
//if we haven't yet created the event, create it now and add it to the dictionary
if (!events.Select(ev => ev.Name).Contains(eventName))
{
currentEvent = new Event { Name = eventName };
//the venues of the new event are still empty
currentEvent.venues = new List<EventInVenue>();
events.Add(currentEvent);
}
else currentEvent = events.Where(ev => ev.Name == eventName).Single();
// the same as above: we have the event now, if the current venue isn't yet on its list, enter it, else use the old one
string venueName = line[1];
EventInVenue currentVenue;
if (!currentEvent.venues.Select(ven => ven.VenueName).Contains(venueName))
{
currentVenue = new EventInVenue { VenueName = venueName };
currentVenue.Dates = new List<EventInstance>();
currentEvent.venues.Add(currentVenue);
}
else currentVenue = currentEvent.venues.Where(ven => ven.VenueName == venueName).Single();
string date = line[2];
string cost = line[3];
EventInstance currentEventInstance = new EventInstance { When = date, Cost = cost };
currentVenue.Dates.Add(currentEventInstance);
}
else
//if the line was too short
unreadable.Add(lineCounter + 1);
}
unreadableLines = unreadable;
return events;
}
}
}

The easiest way to do this would be with PHP and a mySQL database. You can add / overwrite the database with a CSV file, but in the long run, you would be better off developing a simple input form to update the database, rather than going through the task of overwriting / updating the mysql database manually, with the CSV file.

You don't need to start learning PHP to do such simple thing. If you know jQuery a little bit, simply add JSON file to your server, eg. events.json with this structure:
[
{
"event": "Event name",
"description": "description of the event",
"dates": [
{
"date": "20131028",
"place": "Dublin"
}, {
"date": "20131030",
"place": "London"
}
]
}, {
... another event here with the same structure...
}
]
load it with jquery.get, use some templating library (eg. underscore) and make simple template inside the page to display events and details. Finally you will have only 2 pages (or maybe only one), home.html for displaying the list of events and event.html to display details about the event.
Now editing events.json adds and changes events on the home page and details pages. This is just a rough example, it needs to be customized according to your requirements.

Ajax is a technology that alow a conversation between a client side script and a server side one. So in order to use it you will have to study some server side stuff. JQuery is a client-side script which means it only runs on the client machine at the browser.
I would recommend you to start with php it is simplier to learn and to use. And just to read a file is way to easy to learn as you want.

Passing objects from NodeJS to client and then into KnockoutJS viewmodel

So thanks to SO I can pass an object from node to the client, but then getting it into a knockout view model is a bit awkward. These are the steps I have so far (I've included links to the relevant lines as they appear in my github project. Thought the context might help.):
Apply JSON.stringify and pass to the jade file
recipeJSON: JSON.stringify(recipe);
Wrap this in a function in a header script that just parses the JSON and returns the result
script
function getRecipeObject() {
var r = '!{recipeJSON}';
return JSON.parse(r);
}
Call this function and pass the result to a view model constructor
self.recipe = ko.observable(new Recipe(getRecipeObject()));
This works but is there a better way?
Question clarification (Edit): I feel step 2 shouldn't be necessary. Is there a way to directly pass the JSON from node to the Recipe() constructor, without the getRecipeObject() acting as an intermediate step? I tried passing recipeJSON in directly like so
self.recipe = ko.observable(JSON.parse('!{recipeJSON}'));
That doesn't work I think because its not a jade template and has no access to the variable.

According to the answer to this question rendering data into scripts is bad practice and I should instead make an XHR call on page load instead.

Edit
I just saw you linked a github repo! So you're already familiar with most of this...you even have an endpoint set up at /recipe/:id/view, so now I'm really confused...what isn't working out for you? Just the last step of deserialization using ko.utils.*?
Sorry about all the exposition -- I thought this was way more rudimentary than it actually was; I hope no offense taken there!
You really don't want to return a script to execute -- instead, treat this as a DTO: an object that just stores data (no behaviors). An example would be:
{
recipeID: 12,
reviewIDs: [42, 12, 55, 31],
rating: 4.2
recipeName: "A super tasty pie!"
}
This object (representation) is a projection -- a simplified version of the full data stored in the database.
The next step is to create an endpoint to access that data on the server. Let's assume you're using Express:
var app = express();
app.get('/recipes/:recipeID', function(req, res) {
var recipeID = req.params.recipeID;
// It would be cool if this existed, huh?
getRecipeAsync(recipeID, function(recipe) {
res.status(200).json(recipe);
});
});
If you send a GET request to your (hypothetical) application (let's say it's https://localhost:8080/recipes/12), you'll get json representing the (admittedly imaginary) recipe with ID 12.
You can accomplish getting the JSON with jQuery (or any other library that makes XHR nice and pretty)
var recipeID = 12;
$.ajax({
url: "/recipes/" + recipeID,
type: "GET"
}).then(function(recipe) {
console.log("Hey! I got the recipe: %O", recipe);
// Note: you might need to use ko.utils.fromJS(recipe) if the returned
// data is JSON that ISN'T deserialized into an object
var recipeObservable = ko.utils.fromJS(recipe);
});
That's about everything you need to know. Obviously, the devil's in the details, but that's basic idea; let me know if that helps!