Designing Beautiful REST + JSON APIs

Claire:  It is always my great pleasure to introduce Les, who I worked with for many years. This is a very special presentation so I'm excited that you're all here and I'm kinda getting goosebumps. Over 750,000 people have seen Les give this presentation. REST JSON API Design is one of the most complicated things in modern web development and this is, I think, the best presentation on it. So, I'm very excited to introduce Les Hazlewood who is the former co-founder and CTO of Stormpath and now a chief architect at Oktane. 

Les Hazlewood:  Thanks, Claire. Thank you. Can you guys here me okay? Is the volume good? Okay. So, we're here to talk about building REST JSON API, specifically your APIs that you're building and delivering to your customers or your partners, this isn't really gonna be a talk about Octozone API, it's about how you can build APIs, great APIs and how to make them beautiful so that your customers love them. 

Why are we here talking about APIs? Alex's entire presentation, if you were here to see that a second ago, was all about connectivity and enabling support for your developers and your customers and your partners. And how do you scale out your products and make them easier to consume and use across different markets and heterogenous customers? 

So, that's really what APIs are all about. It's about connectivity. But why are we here talking about REST specifically? So, REST was invented by Dr. Roy Fielding as a PhD thesis for his computer science thesis. And he specifically lists out these six points of why REST as paradigm, is very valuable. And we don't really have a whole lot of time to go into each individual point but, I really recommend that you check out his thesis. He goes into these at length. But all of these, really, bundle up to ubiquity. 

How do you get massive internet scale adoption that people across industries, markets, verticals, can all communicate with each other at massive scale? That's what REST is all about. But why are we here talking about JSON? This is a REST plus JSON specific presentation and the answer is, that JSON is pretty much everywhere. Regarding of programming language, it's supported pretty much everywhere. It's simple, it's easy to read for humans. So, this is the lingua franca of the modern web. 

If we're here talking about the modern web and JSON and in HTTP, this all must be easy, right? These technologies, JSON and HTP have been around for 20, 30 years or more. So, this all must very simple. We've nailed it. Right? We got these technologies down. But the answer is that, it's not really easy. It's really, really hard actually if you're a provider of a REST API. You're building a REST API and you're presenting it. And the reason why, there is no specification for REST. It is an architectural set of guidelines or principles or a philosophy from Fielding's thesis but, it's not codified into a set of steps. Do one, two, three and then section A and all these other rules. It's not codified. 

So, we're left to interpret his specification and different people interpret things differently. However, REST can be easy if you follow a set of common guidelines and we're really here today to talk about the fundamentals of REST and what those guidelines are and what they should be. There is a huge amount of technical data and questions I'm sure you guys would have on some of the minutia and intricacies of REST and unfortunately we don't have a lot of time to talk about that today but, I will be available after this talk, hosting a Birds of a Feather luncheon or what one of the lunch tables. So, feel free to come talk and geek out about that kind of stuff as well as Claire said in the developer lounge from 1:30 to 2:30. 

In the context of REST, a lot of times you'll hear this term 'HATEOAS'. Hyper media as the engine of application state. This is a huge mouthful and it's a terrible acronym but what this really is trying to represent is that everything in REST API should be in the documents exchange between client and server. And there should be nothing else other than what's in the document that explains everything. So, if you think of a web browser, accessing an HTML document. Everything the browser needs in order to render the page, in order to trigger behavior, in order to jump from one page to the next is all codified in that HTML document. Right? 

So, that HTML document is fully RESTful because the browser or person interacting with the browser doesn't need to go to some website or re documentation to figure out what to do with that HTML document. So, when you hear 'HATEOAS', just think, everything is contained within the document. Everything that's necessary is in the document. And what this really boils down to, if you dig into this a little further is it all boils down to two things, links and state transitions. 

Meaning, how do I like from one document to another? That is the huge power of the web that you can link from a documents from Google, to Stack Overflow, to your company's website. The power of the web is all about linking. How do I get from one location to another? State transitions is not, how do I just get from one location to another? It's, how do I send data from one location to another? So, I'm not just getting a resource or I'm not just getting a new document. I'm able to send data to a separate location from one server to another, from one website to another. 

So, HATEOAS boils down to links and state transitions. In fact, Fielding's thesis, if you've heard him talk or you've seen some of his blog articles, everything in REST boils down to these two concepts. But a lot of people misinterpret Fielding's thesis and so, he's actually come out with a blog article, it's the URL at the top. And this presentation will be online so you can check it out later, where he clarifies some things that are kind of shocking, especially if maybe like me, you come from a type safe programming language background where things are concretely defined and statically typed. 

But Fielding specifically says ... Well, I'll get into the typing in a second. One of the first things is, communication protocol independent. So, you don't want to couple yourself to the communication's protocol. So, if I have a RESTful API, it should be able to work whether I'm using HTTP as the protocol or if I'm using raw TCP sockets. I should be able to take my RESTful document and send it over to TCP or HTTP. 

And so, if you find yourself building a REST API and you're hard coding or coupling yourself to HTTP headers and status codes, you might not be as RESTful if you think you are because if you try to translate to a different protocol, it would break. And so, you're violating one of Fielding's principles. 

They're also media type centric. Meaning, it's all about the documents and what's in the documents. And the type of documents and how to pars it is codified by an internet media type or mind types. So, REST is all about media types and how you pars documents. There's also no fixed names or hierarchies. This is really hard for people to understand and in fairness and in truth, it took me a couple years to wrap my head around this. Meaning, if I go to a particular URL in a REST API that says '/user/123' as a human, I always expect, every time I interact with that URL, that I'm gonna get a user resource back. 

REST is not about that. REST is, I have to go hit that URL, get the data back and I have to pars it to see if I can make sense of it. It could be a use today, it could be a car tomorrow. It sounds weird. As humans, we make sense of URLs but in reality, URLs and naming and hierarchies, like hierarchal URLs with name spaces mean absolutely nothing with REST and so, if you couple yourself to this notion that thing shave to be types according to the URLs, you're actually fundamentally violating REST principles. 

To that end, everything in rash should be dynamically typed. Meaning, I can't interact with a resource and expect I'm gonna get a user back every single time. I have to pars the document and everything in it before I can say, "Yes, this is what I understand as a resource for a user or not". 

So, there's no static typing's in REST. You can't always hit a URL and expect to get the same thing back every time. And if you do or if you code your clients to just blindly access properties or fields then you're violating REST. That's not RESTful. 

And most importantly, and I'll just reiterate again is zero autobahn knowledge. Meaning, everything that should be necessary for our client and server to work with each other is 100% contained in the documents exchanged between the servers. Not necessarily in HTTP headers, not in some other mechanism. Everything's in the document. Which means, you shouldn't have to go read documentation to figure out how REST API works. If you're reading docs, it means that the information is not in the doc that's being exchanged. 

Really difficult to address and understand but again, happy to talk about these things after the presentation. So, if Fielding comes up with all these clarifications on his Spec, how do we meet these requirements? 

So, we really want to break things down in some fundamentals. REST is primarily, given that it's an acronym, Representational State Transfers. State often translates to nouns. Nouns or resources are the important thing of what's being transferred. There's not verbs represented here. Like, get a user, delete a user, that kind of thing. It's really just, user of account or company or organization. They're all about the nouns. 

And they should be coarse grained not fine grained. Meaning, everything that's relevant for that particular resource should be in the resource. You shouldn't have to necessarily jump to five or ten different links or other documents to get everything you need about that one particular resource. And that's important because the more information you have in the resource or the document for that particular resource, the more clients can self program and satisfy requirements that maybe you never though existed. 

So, instead of you creating links to all these other things around say, for example, user object, if you have all the data in there, the user or the client, can consume that data and do things with it that you maybe never envisioned. And that helps with Fieldings support for ubiquity, or large scale adoption. 

So, to illustrate what I mean by nouns versus verbs, what if I had some URIs in an application that I'm building like this. Get user, I want to create an application resource, maybe I'm gonna update a particular group and it's data. Maybe I want to verify a users email address. This looks fairly innocuous. Not too bad, just four URLs. But the reality is that, when you couple verbs with nouns in your app, this quickly explodes. Right? 

And so, you're gonna get a ton of URLs in your app that look like this. You're gonna have to support them all indefinitely and it's gonna be a huge pain in the butt. If you've ever done some SOAP or RPC programming in the 90s, early 2000s, this smells like bad RPC. Don't do this. This is a bad, bad anti pattern, it's gonna cause a lot pain. So, if we can't or shouldn't do these things, what's the answer? Well, we want to keep it really simple and to do that, you can categorize or break down resources into two particular types. Collection resource, meaning you have collection of things and instance resource. Meaning, one individual thing. 

Now, conventionally collection resources on the web are typically named in the plural, not the singular because it always ... it's self documenting, to a human at least, that you potentially interacting with more than one thing. So, it's not application, it's applications. It's not user, it's users. We recommend that you stick to this kind of convention, it just makes things a lot easier to read and understand. 

And so, if this is a collection, an instance typically hangs off of a collection. It's an easy way to resource that particular resource within the particular name space. Now again, name spaces don't matter. This could just be junk, gobbledy gook, right? As far as clients and servers in the REST [inaudible 00:12:10] paradigm are concerned, but this makes it easier for humans to understand what's going on. So, it's a nice benefit. 

So, these are nouns and how to interact with these nouns, these resources. Well, what about behavior? And if you're using HTTP as a protocol, the answer is HTTP already has behavior built into the spec in the form of HTTP methods. And so, you can do a get put, post delete, head options, you've probably seen these before. But one thing that's kind of confusing at times is that, a lot of times people think get, put, post and delete have one-to-one correlations with create, read, update, delete. We've all heard this crud term for data base storage and stuff and everyone ... Not everyone but a lot of people think that post, get, put and delete of HTTP will always have a one-to-one correlation with crud. 

And this is actually not the case. This is where HTTP, the specification actually allows you to do multiple things sometimes. However, as you might expect, get really is a read. It is a one-to-one correlation. Delete is really delete. I'm trying to delete something on the server. Head is a meta data operation. I just want to get some information back but I don't actually want the resource. And options is HTTP communication options. Preflight request for COORS and things like that. 

Again, these are all HTTP specific concepts. They give you some niceties on top of RESTful documents. But 'put' and 'post' are not so obvious. They can both be used for create and update. So, here's how. You can use 'put' for create if the identifier of the resource, meaning the fully qualified or canonical URL of that resource is already known to the client before they submit the request. 

So, if you allow your clients to create their own identifiers for URLs, then you can use 'put' for creation. In this case, somebody's creating the application and they've just generated their own ID for their own app and so, they want to save it under that identifier. This isn't too common but it is totally plausible or possible with 'put'. 

You can also use 'put' for update though. So, in a full replacement or rather, in a replacement operation, I can put to that same URL, same existing ID and I have two properties. In this case, name and description. But the implication here, kind of the interesting bit about 'put', this has to be a full replacement operation. Same with 'create'. You have to specific 100% of the data for the resource. You can't use partial updates or partial creations. And the reason for this is that 'put' is idempotent. It's mandated by the HTTP spec. This is not even a REST thing, it's an HTTP thing. 

That every 'put' re-question has to be idempotent. Meaning, you can send the same request one, ten or 100 times and the state of the resource after each one of those requests is gonna be identical after that's done. So, if you send in one property on request one and a separate property one request three and then that first property is sent in on request 15, the state of that resource is gonna be different across those requests. 

That violates the HTTP mandate for idempotency. So, 'put' has to be everything, the whole resource. If I'm gonna send data in on a 'put', the entire resource and it's full representation has to go. So, just be careful with that if you use 'put'. 

Post can also be used as create as well. Typically, it's put down on a ... Or rather 'put' ... or sorry, 'post' is sent to a resource, maybe a collection resource, that's the most common thing. And in this case, I'm just sending a name, property and the post response sends back two HTTP headers, location header and a 201 created. 

So, this is really important for HTTP because 201 means not only was the request successful but it was successful and I created something as a result. And you have to set the location header to tell the client, "Hey, this is where this thing that we just created, it resides here. You can go interact with it further if you want to." 

And this is really important because if you look in the original requests, the client has no idea where the canonical location is. It doesn't know what the identifier is. So, you have to set a location header and make sure it's 201 not 200. 

Post can also be used as an update. So, in this case, I'm sending in a new name. I just want to change the name field on that particular application resource. In this case, all you need to do is return 200 okay. Because the client already knows the location, already knows the URI and 201 is not necessary because you didn't create anything. You're just changing the state of something. So, 200 okay is totally acceptable. 

But as you notice in these examples, we're only sending in one property or just partial properties and the reason why that's legal is post is not idempotent. It's the only HTTP method in the spec that doesn't guarantee idempotency. 

And that's really important to understand. Basically ... and the HTTP spec is very vague on this. It basically says, "It's a server processing directive." And that's basically the extent of what they say. Which means that you can do whatever the heck you want. You can mutate the state, you can have side effects and change state elsewhere. You can do whatever you want during the post. So, partial data, partial submissions totally fine. 

So, we've talked about nouns and verbs. The other part is just very briefly a review of media types. And a media type ... And this took me a little while to wrap my head around earlier in my career but a media type is a format specification in the set of parsing rules associated with it. 

How do I read the document and make sense of certain fields? Media types are not schemas though. Right? It's not saying, a user resource has this property and this property. A media type is more about, here's a document and I can read properties. It tells you how to pars the information, not the mandatory structure of that information. If that makes sense. So, keep that in mind and clients and servers negotiate documents via an HTTP with the accept header and the content type header. 

So, a client will accept the accept header to say, "Hey, here's all the media types that I support, that I know how to pars." And the server, in a reply or a response will set the content type header that says, "Okay, you've told me what you supported. Well, the content type's gonna reflect what I'm actually sending back as a server to the client." 

So, that's called 'content type negotiation' between clients and servers. And the content type is, most of them are standardized but you can also have subtypes. And so, if I have application/JSON, which probably most everyone here has seen, you can actually have further specifications on top of JSON by using this prefix notation, this ion plus JSON... And you can have different JSON specific specs based on this plus notation. 

And we'll be talking about ion in a second but this is just an example of a media type that's not just JSON but it's JSON plus an additional rules or some structure. So, okay, let's talk about design you guys are really curious about design and jumping into what an API could look like. How do we do HATEOAS cleanly in JSON? There are no formal official final specs but there are a set of competing specifications on the internet and ITF and the W3C. And I'm gonna talk to you today about one called Ion. Ion. 

In all fairness and just being forward, I'm the spec lead for this particular initiative but we have a whole set of people that are participating in this effort from different companies, different corporations. So, it's not just an Oktane led thing, it's across company led thing. 

But we think ion addresses some HTTP or RESTful concepts in JSON that others maybe don't address too well. So, what does ion look like? How would you use it in an API? So, everything in ion is just JSON, right? It's ion plus JSON. So, that means that normal objects are parsed as normal objects. There's nothing special, no surprises here. So, this is just a simple object that represents, maybe a person. First name, last name, birthday. 

Ion also introduces notion of a value object. So, it'll have an object with a value property. And that's the only thing that's required to have a value object. Now, why is a value object relevant or valuable in JSON? Why don't I just have properties? Here's an example. So, let's say I have a greeting field or member and in this case, the value is 'hola'. 

Which most of us will probably recognize as a Spanish national language word but a machine has no idea what that means. A machine doesn't know that this is Spanish. It doesn't know if it's english or anything. It just knows that, "Hey, here's a value to this property." Well, if you replace this with a concept of a value object. So, now I have greeting that's assigned to a JSON object with a value property, it allows me to add additional metadata associated with that value. 

So, in this case, the value is still hola but now I have a laying parameter or member called es to represent it, it's a Spanish national language word. So, the point of a value object is not only do you just have a value but you have all this space for extra meta data to prevent or provide relevance for clients and humans too. They find value in this. 

So, that was a single resource. What abut collections? You might just assume, okay well I have a value field. Let me just stick an array in there. This is a naïve implementation in that there's some challenges or difficulties with this. And so, ion doesn't represent collections this way, it actually wraps it as a value object. So, it's an object with a value property. And the reason why is again, you can add meta data. 

So, everything that's in the value array, maybe there's metadata that's relevant to each element in the ray. And you don't 'want to repeat it on every single element in the array. Or maybe there's things like length or max size or things like that that are only relevant to the collection itself, not each independent element in the collection. 

And so, ion collections are value objects that are wrapped or rather a value field that's wrapped in an object. And we can take this construct and start addressing things like linking and state transitions, right? Things that Fieldings says is really important for REST APIs. So, if you remember or rather HREFS in HTML pages, a HREF for the anchor. Disrupted in hyper media is paramount. This is one of the most important things that Fielding says and it all boils down to links, canonical URLs and state transitions. 

It's important to know that HREFS or URLs, they have IDs in them but IDs in a URL are completely irrelevant to a client. All it needs is a URL to go access and that's the only thing that's necessary. Now, as humans we imbed IDs and URLs for our own cognitive ability or understanding but, clients don't need this. So, IDs exist but really, the ID of the web is a URL. That's the thing that canonically identifies any particular resource. And it's critical for linking, as we'll see. 

So, how do we do it? XML has it in the form of the XLink specification. JSON doesn't have it yet. Now, there's some competing standards. Ion is one of which, HAL is another. We'll talk about some of these in a little bit. So, how do we do this? How do we make it simple and clean in JSON? 

Well, ion does it by just having a simple HREF property. It's an object with an HREF field with a fully qualified URI ... Or it doesn't have to be fully qualified. But, it's just a URI that the client can negotiate. And so, if you have a link like this, it's very simple, it's pretty easy to understand. If you've ever done HTML this is totally natural and intuitive. 

So, if I can use this link, how do I link between resources? So, here's an example. I'm gonna get a particular user resource and a user in this resource has a given name and a surname but how do I link from that user resource to the company? Maybe their employer? And so, the answer is, you just use that same ion link concept where if I have company that's referencing this object with an HREF field ... So, this is pretty obvious. This is clearly a link as is recognized by us, if the media pars type recognizes it as a link, it can traverse it and go to that next resource. 

What about collections though? This is a single resource. How do I link to a different collection? And that's not really any different. So, let's say if a user ... And they have a collection of groups that they're a member of, you just link to it in the same way. It's still an HREF, there's no difference. However, this isn't quite enough information to tell a client exactly what's going on here. 

We already have this notion of link relations to find in the XLink specification. Like, what is the purpose of the link? Why does it exist? And so, we just continue that here in ion as well. So, we have a RELL link relation. And this is no different than if you've looked at the HTML anchors, they have A, HREF, REL, some value. Same exact thing just in JSON notation. So, one of the design considerations for ion is to be incredibly intuitive and easy to understand and also to support transliteration between HTML and JSON if that's a need. Sometimes that's valuable for legacy products or existing documents that need to be translated to JSON for modern clients like Angular React. 

So, the client now knows. Not only is this a link but it's a link to a collection of things. And then, of course, you can add more metadata in there to give ... What's the media type? What media types are supported so I know maybe I can negotiate and get XML if I wanted to or JSON? We also recommend this notion of a self link for every resource that's on the web. So, it's a link like any other, it's nothing special. But one of the cool things about ion is, even if I don't specify a REL parameter, the link relation is automatically assumed to be the name of the field. 

So, in this case, self is the link relation. And self is a pre approved IANA link relation but it doesn't have to be IANA approved. If your application understands different ones, you can name this whatever you want as long as the client and the server can speak the same language. 

Very simple, easy and intuitive and it supports HTML transliteration quite nicely. So, that's ions approach. What about HAL? Sometimes people ask about this. HAL is another linking technique, if you've never seen it on the web. Of course I'm biased but a lot of people think or at least those on the ion spec committee, that HAL kind of fall short in a lot of areas. It does support links but it forces you to create a property in your object model [inaudible 00:26:45] with a bunch of other things to outside locations. 

But that's not really contextually relevant, is it? So, for example, if I'm creating an HTML document, when was the last time HTML forced you to pull all your links our and stick them into an HTML element called 'links'? That's absurd. Links are most relevant when they're next to or combined with the content that's directly associated with that link. 

And so, HAL forces you to change your object structure whereas ion allows you to just put it wherever it's most contextually relevant. So, it's again, much more like HTML ion. So, it allows transliteration by design and we're trying to keep things simple and easy to understand. We don't want you to remove the context from your document or move it in different locations for the benefit of a parser. Context matters. 

So, that's linking. If we're talking about linking in state transitions, what does a state transition look like? I can get from one document to another, from one resource to another. How do I change state from one location to another? How do I things like creations and updates? 

So, you remember HATEOAS? The rules in HATEOAS about things being in a document? Most of us today, and all of the APIs this is, to be fair, this is true of Stormpath, it's true of Oktane, a lot of people don't 'do this but the answer to most developers today or the most take on is, let me go look at the docs and I'm gonna figure out how to do that post. 

We've all done this. We've gone to the documentation, developer documentation and we look at the URL and all the fields and properties that it supports and then we're like, "Okay, let me go code that and I'll code within HTTP get and maybe ill use CURL to test it out." This is not HATEOAS. If you have to go read docs in order to figure out how to interact with an API, that API is failing. It is not RESTful by definition. Fielding's specifically says, If you force this on your users, he does not want you calling your API a REST API. 

Call it an HTTP API or some other kind of API but don't call it RESTful. So, how do we get around this or how do we address this? How do browsers work? Browsers can submit state from one location to another. How do they do it? They do it with forms. This is nothing new or special. We've been using this for decades with HTML. 

If I need to submit state or change state or from one location to another, I have an HTML link or rather a form with a target location and a bunch of form fields. So, what does this look like in ions? If forms are so natural and intuitive, how do we leverage them in JSON? Here's an example, this is a form and if you look closely, it's just a link. There's an HREF property. There's a REL property that tells you specifically, in this case it's a 'create' form. There's a value, that's a collection. We've already seen collection value objects before. There's one addition in this case for HTTP support and there's a method property that says specifically it's gonna be a 'post'. And even this is optional. 

If you don't have the method property, post is assumed by default. And it's got an array of some fields. Name, login, [inaudible 00:29:56]. Stuff, this looks very much like an HTML form. So, what is a form, exactly? If we break it down, what does it look like? It's really most importantly, a collection of form fields. There is a set of fields that are represented to somebody. Either a user or a machine client that represents the state, the information that needs to be transferred. 

So, each field, also has a type, a set of validations associated with it or constraints. There's also a destination IRI or URI of where to send that information on a HTTP transfer. There's also a method requirement, at least if you're using HTTP as protocol. Now, if you're using a different protocol like TCP, that method property doesn't need to be there. You can use some other mechanisms of identifying updates or creates. 

Ion already has a collection concept. We've already seen in the link, a destination IRI. All we need is a REL to tell the client what type of link this is and these are actually pulled from IN as well. Well, except for form but edit form, create form, query form, these are already IANA link relations. IANA already define us, they know how valuable this is and it's been around for years. 

You can even make things way more complex and more rich. You can have link to create a user and maybe I have a collection or visited continents and I want them to specify which continents have you visited? You're gonna have constraints like, you have to choose one but there's no more than seven. Here's all the options. This looks like an HTML form. This is not a surprise to anybody.

But if I get this in a document, I don't have to go read docs. I know exactly what's supported. I know what the boundary conditions are. I know what the permutations of the data can support. Everything's 100% represented in the document. I don't need to read documentation. Search also works too. So, yes you're getting information but you have to send information into that URI in order to customize how those results are sent out. So, you just use the query form link relation. 

And I this case, this'll translate or serialize to a URL with query parameters. Users, question mark, username equals blah, email equals blah. Serialization from JSON to URLs is trivial. It's been defined in HTML for a long time. So, if I have forms that represent structure and data and requirements and constraints, sometimes we also get this question of, what about JSON'S schema? 

Our take on this is that it's not needed. If you've seen the forms example. JSON'S schema is about defining what fields and constraints are available on a resource. What's really important to understand that REST is not equal to RDBMS as far as a paradigms. Right? Create, update, delete are not equal to crud. I'm sorry, get, put, post and delete are not equal to crud. And to illustrate this, do browsers need schemas of resource types? No. They only have one that they have to worry about and that's the HTML specification that tells you the structure of elements in the doc. 

But there's no HTML spec that says, "This is my HMTL user resource. This is my HTML group resource." JSON'S schema is just ... A lot of us think it's extraneous and not necessary. Because forms do the exact same thing, right? They show structure, they show constraints, they show what you're allowed to do. And they're also more powerful. And to that end ... Or as an example, what about forms that accept all this information, but you might have hidden form fields. Maybe a JWT to retain private state across when you access that URL, when you send that state from one location to another? 

JSON schema, if it's representing a user or an account, it's not gonna have a field that's representing for hidden state or field state or client side specific concerns. Forms do everything JSON schema does and more. And according to Fieldings rule about dynamic typing, they're fully dynamic. I can access a field, or sorry, a form on one request and I can go back to that resource and the form might've changed maybe some state or information has required additional fields to be entered or maybe they're missing because they're no longer required. 

JSON'S schema doesn't really reflect that. No, in truth you can dynamically generate a JSON schema document to represent all these permutations but, why go through that effort? Forms already do that for us. So, just keep dynamic typing in mind. Things can change and they should be able to change. Clients should always be able to go to new redirect locations and if you move a property out of your REST API tomorrow, it shouldn't break a client because the client shouldn't hard code that that property will always exist. 

Forms tell you the structure of what's necessary. So, what about base URLs and versioning and date time representations and pagination and async operations and long lived ops and batches and clean and airs and security and all this other stuff. This is all the really good stuff that I'm sure you guys would love to dig into and unfortunately, I can't dig into this in a 45 minute presentation but I would really love for you to participate in the Birds of a Feather luncheon if you're up for it. Again, I'll be available from 1:30 to 2:30 after that. 

If you're inteRESTed in ion and you want to see it or try it out or understand the spec, we're actually pushing to become an IETFRFC. So, we're gonna declare a formal RFC that represents this, that codifies this to be a fully comprehensive media type that supports all of the HATEOAS concerns that Fielding talks about. 

Until we become and IETFRFC, you can check out the specification online at ionwg.org for ion working group and the spec is right there. You'll see everything. We have one thing that we've not yet addressed in ion and that is the notion of custom constraints. So, let's say that you have a form field and you want to, I don't know, you want the values to only be a certain color. You want to create a color type for your form field. And maybe that color type only accepts 16 different string [inaudible 00:36:20]. Blue, green, red, whatever. Right? 

Ion doesn't have a native color type defined, just like HTML doesn't. But it would be really convenient for you to be able to define your own set of constraints. Basically saying, "Hey, this is a string field. Here are the regular expressions or set of properties. And this or these values". That's basically custom constraints definitions and so, this is the last thing we have to finishing and we're working on that now. But as soon as that's done, we'll be able to go the IETF with what we feel is a comprehensive spec that covers all these cases. 

To that end, I've got a little bit of time left for questions. Feel free to chat with me today at the lounge. Again, the Birds of a Feather lunch right afterwards. Thanks for your time. I'm happy to open it up to any questions. 

Claire:  Any questions? 

Audience 1:  So, you said that forms are better than JSON schema and HTML doesn't really need schema but what about the differences between the HTML 4.01 transitional strict, HTML5? Are there aspects of that that you foresee being able to add two forms? Like, through versioning? Not saying that JSON schema is necessary because it does seem that forms is robust enough to take its place but is there natural definitions of versioning's within that you're expecting to have built in? 

Les Hazlewood:  Natural definitions of versions for the structure of a document, is basically what you're talking about. 

Audience 1:  Yes. 

Les Hazlewood:  So, yeah. So, HTML does have different schemas for the media type itself. Or rather, for the HTML specs. So, there's HTML 1.0, 1.1, transitional strict, right? You talked about these. Those represent the structure of what elements are allowed within an HTML document for all browsers. So, they couldn't changed the media type to applications/HTML colon version equals one. But they also represent it as a header that you can tell what kind of version it is. 

But those versions are about the structure of the document or the media type itself that has nothing to do with versions of the document that you're presenting in a request or a response. So, you can have HTML 1.0 document for one URL and then you can show 1.1 the next and depending on negotiation between the client and the server, that structure can change. But it's not application structure. It's not, "I need a new version not represent a user resource or an account resource or a company", you know what I mean? 

So, that versioning is really around the media type itself as opposed to the content within the media type, if that makes sense. And so, forms allow you that dynamicity if you change your own applications content model and content structure. The parser doesn't need to know about any of that stuff. So, HTML versions and schema versions are all about the parsing of it. Does my parser, does my browser need to change to look for these new elements? 

Google and Microsoft and all these websites still work because the browser can read those different schema changes. But you can change your page from one day, like we do. Our developer platform, developer.octa.com changed dramatically this week. But your browser didn't need to change because it already knew how to parse those elements. Even though the material content was very different, if that makes sense. 

So, forms allow you to do that. To that end, if things in ion need to change or versioning is necessary, you'll have things like ion version 1.1 or 1.2 or 2.0 but, you don't need a version field to represent your document state, if that makes sense. Now, you can add it as part of your self link, right? That's your meta data for the page. You can add that to give your own application context that, "Hey, this is version one of a user versus version two" and it's in the doc so that's good. But the parser doesn't need to know about that because that's application level concerns. 

Claire:  Other questions? 

Audience 2:  So, I know this totally breaks HATEOAS but, it makes sense to me to have the authentication and authorization in the headers and I'm still struggling to think of how the jots and the tokens can reside in the links when they're temporal. And so, that authentication and authorization may not be accurate if you try to use that link a half an hour from now. 

Les Hazlewood:  Yeah. So, if you only use HP headers and it's not in the document, then it's not really HATEOAS compliant. And so, you're just, you're coupling yourself to HTTP, which may be totally fine. It's just, violates the fundamentals that Fielding was talking about. 

Audience 2:  So, how, in HATEOAS, do you ... 

Les Hazlewood:  So, the rule for HATEOAS is, have it in the doc. There's nothing that prevents me from creating a document with a link to an authentication link that has a URL with an embedded JWT and I can click it and it'll take me to some page. Or I can represent a form of all the things that are needed to be submitted to an authentication end point. 

And so, for example, maybe I hit /user/123 and it this is why dynamicity is so important. Maybe I hit that and if I'm being fully HATEOAS, it doesn't return me a user object, it returns me an authentication form instead. And then I fill it out and then I submit and then it'll return me the actual user. If that makes sense. 

Now, you can change the URLs for human empathy. You don't want to necessarily confuse people but everything I just talked about is still represented as a link or a form in the document. HP header wasn't necessary at all. Now, if you're leveraging HTTP clients to build STKs, for example, it's kind of nine to use those HP mechanisms because it's codified, there's standards for it. It's just not HATEOAS and so, what I recommend is if you're ever conflicted about, am I being HATEOAS compliant or compatible, if you stuck stuff in HTTP headers, add it in the document too and then you're covered. 

Clients also, depending on their complexity or their implementation, may or may not have direct access to HTTP headers at the time that their document was parsed. A lot of times, you'll have request or response pipelines and they'll strip the document out and leave the HTTP stuff away and send the documents somewhere else, right? Well, if I had that document, out of the HTTP context, then anything that was a header, I can't leverage when I'm looking at the document. 

So, if you're ever conflicted, just repeat it in the document. And that should never be a problem because clients shouldn't assume that it's always gonna be there. They should inspect the document. Sometimes the link might be there, maybe other times with no authenticator, it's not gonna be there at all. And so, dynamicity, parsing the resource, not assuming what fields are available, is super key to HATEOAS and it's really important for ubiquity. 

So, you can hard code fields, leave them in the resource but I promise you, every time I've ever seen that, my own APIs included, it becomes brittle, customers break their applications because they're just assuming that field's always gonna be there. It is not good. So, having STK that's dynamic, that says, "Here's the fields that are available but I don't know what they are. You have to make sense of them" is way more flexible and will allow ubiquity across heterogeneous domains and markets and verticals and stuff. 

Speaker 1:  I have a question. Pass this one down for me. 

Audience 3:  In terms of client adopting of HATEOAS, have you seen a lot of people actually, like external parties, implement HATEOAS and follow it or do they tend more to see what the doc structure is, read the docs and then hard code it and just ignore [crosstalk 00:44:19]. 

Les Hazlewood:  The vast majority of REST APIs today require you to read docs to figure out how to interact with them. And it's very unfortunate. However, there've been a lot of people keen to this problem and things have been changing pretty well the last couple of years. So, for example, have you guys ever received an email from Travelocity or Kayak.com or whatever and you get that email in your inbox and Google, in it's Gmail reader, shows you the departure date that the landing date and all this other stuff and you're like, "Dang, how did Google figure that out, man? That's so cool and it can add it to my calender automatically." 

The answer is that that information was codified in the SMTP email that was sent to you as state in the document and they also use something called 'JSON LD', which I don't have time to talk about today, that gives machine clients contextual knowledge of what's in the document. So, if I have a user name field, a machine doesn't know what a user name is. You might as well just call it a bunch of random characters. It has no clue what a user name really is. 

But JSON-LD will attach context to that user name field and say, "This field represents what schema.org's definition of a username is." And so, the machine client now knows, "Oh, I found this field that has a context of username attached to it? Okay, I can treat it as a username. Because Microsoft might call it a login, you might call it a username, other people might call it something else but JSON-LD allows a client to look at any of those fields and know, "Oh, this is a username even though they call it a login. This is a username even though they call it a login. Oh, this is a username even though they call it username." If that makes sense. 

So, HATEOAS is getting much better and contextual relevance around JSON-LD is getting much better and so, really cool stuff like what that email example from travel sites is all about HATEOAS. And so, people are seeing the huge benefits of that and moving forward. And the biggest benefit, I think, is that regardless of what domain you're in, I can interact with any third part whether it's Microsoft, google, Facebook or whatever and I can understand that document even though I've never coded or I don't have a Facebook native STK because Facebook is giving me the context in the document, if that makes sense. 

So, things are changing for the better and much more so in the last couple years. 

Claire:  Alright. More questions? Okay. 

Audience 4:  Can I just suggest that breaking [inaudible 00:46:40] is not really awesome idea that HATEOAS, when you guarantee that a URL represents a known thing break [inaudible 00:46:50] across the entire internet. 

Les Hazlewood:  Wait, so HATEOAS can't guarantee a URL and what about the caching? 

Audience 4:  If you can't guarantee that the content of a URL is the same as what it was the last time that you requested it then you can't cache. 

Les Hazlewood:  You can cache it, it just dependS on what semantics you associate with the caching, right? So, you can set a header or represent in the document, "Hey, this is cache able and here's an expires header or an expires member in the document that sys it's cache able for this long". Like, we don't change it or it's not gonna be that ... The relevance won't matter for a certain amount of time. 

If your API is so dynamic such that things can change dramatically across resources, that just means you won't enable caching for that resource or that end point if that makes sense. So, caching is totally plausible and feasible, it just depends on what context you give to the cache and how long you're saying it's allowed to be cached for by the client. So, that's not precluded at all with HATEOAS, it just depends on how dynamic do you want to be? Are you gonna constrain yourself to not change that doc or are you gonna tell the client, "You can't cache anything because I might change at a moments notice"?

Either of those are completely legal, that's a choice you have to make based off your application. Thanks. I'm out of time at the moment. But, thanks for coming in. Again, I'm going to the luncheon or the Birds of a Feather, feel free to join me there. Otherwise, take care and have an awesome afternoon.