About this talk
OpenAPI Specification, formerly known as Swagger, is a well established standard for defining REST API interfaces. In this talk we'll quickly go over Swagger, how to integrate it with Node.js, and a few best practices to deliver robust enterprise class API service.
Title of today's talk is, "I've got swagger, have you?" I'm Tanzim. I'm lead architect of the company called Starcount. We deal with big data and stuff like that. Before that I was an embed programmer. Lots of time spent developing on phones at Nokia and Symbian. So what is swagger? I mean, I guess you guys already what it means literally, but in this particular context not necessarily moves or anything like that. Swagger is actually a standard for REST API. That's as simple as that. It's a standard that you use to define how your REST APIs look like and work like. I think, the people who developed Swagger realized at some point that if this wants to be taken seriously and needs to grow up, it probably needs to be called something else. So the official name for Swagger is actually OpenAPI Specification. It's an initiative under Linux Foundation, and it's chaired by some of the big names in the industry like IBM, Google, Microsoft, Paypal, and a bunch of other guys. The current version is actually 2.0. Just like OAuth no one wants to remember version one. Version three is in active development. You can check out the whole governance model and the specification itself on the OpenAPI OAI website. Before we go to why swagger and things like that...sorry, into details, we basically. Let's talk about why we actually need something like swagger. All right, because standards are actually generally good to have unless you end up in that classic ex-scarcity scenario where situation you've got 14, 15, 20 multiple competing standards and you need to introduce another standard And just end up adding another on top and confusing the hell out of everyone. Thankfully, the situation with REST interfaces in terms of standard isn't that bad. I actually saw someone talking about API blueprints in the next LNUG which is another standard for defining specification, but it is generally accepted that OpenAPI Specification is the most popular one and lot of the people on the enterprise has been pushing for it. One of the reasons, they didn't take off very well hourly was 1.0 like, I said no one wants to talks about, it wasn't that great. But 2.0 has addressed most of the problems, and three's going to be amazing. Some random guy said, "You don't stack overflow." By standardizing how your API is defined, Swagger actually helps do a few things which makes sense like API documentation. Who wants to write that stuff by hand? It should be generated by machines and things like that. Do you really want to do documentation for what types, what parameters, how validations work? Also consumption is important. When you think about APIs you think about you're writing bunch of codes, plan code against it, but sometimes it could be another machine who's consuming your API and they could be really good if the specification is machine readable. And then of course, the whole story around tooling and automation. If you can have specifications then you can start building things around it because there is standardized mechanisms of dealing with these stuff. So what's the Swagger advantage? Well, like I said before open specification with open governance model. It may not sound that important, but it actually is. People who have dealt with Bluetooth like me know this. Builds on no nonsense technology, so uses JSON Schema, JSON, and YAML under the hood which we'll see shortly. There's wide adaption for it, like I said, now that API services are becoming popular, you've got things like Amazon has API Gateway, Azure has Managed API, and you've got turf by the companies likes Apigee which actually got bought by Google who was one of the original proponents of Swagger. So they have all got financial support for importing Swagger definitions and working with Swagger definitions to define your AP interfaces and your infrastructure for REST as a service. So let's Swagger. By the way, Bill Gates did not say that. Let's take the simplest API you can think of. Let's call it, hello API. The objective is your API is running on your machine locally, and it's got a top level endpoint called hello, and if you do a get with a parameter name, it will send you back a message in JSON that basically would be hello comma whatever you specified in the name. It couldn't get really simpler than that. So if you have an API and you need to basically define this or describe this as a swagger specification, what would you do? You'd write a lot of stuff, but it's actually not as bad as it looks. So start off, you start with Swagger and you specify version number...but I just find that "Jamal" is a little bit easier to reconcile with when you're looking into something like this without way too many braces and quotation marks. So Swagger basically...every Swagger documents start with the version number because version number is important. Then you've got an information section that talks about your API itself, what is the version of that API, where I would you find that API. Things like base paths and schemas or schemes are all part and parcel of the game, where does the actual API endpoint start from, which path in that host, what are the schemes that it supports, and what type of content will it send it , send to the client, and what type of content it can actually consume. For example, in this one it's application/json. It could have been XML and whatever format, and it could have been multiple also if your API supports multiple content types. And then of course, consumes and produces. And then the most interesting which is the path section of it. Remember, our API is hello. So there is a path in there that says, "hello," and it says, "get" and you can do all known HTTP operations on a Swagger ednpoints. So it could have been get, pause, boot, patch, delete, but because we are using get so I've defined get over there. And as you can see it's pretty straight forward. It's got a description, it's got an operation ID. Operation ID is a bit of an interesting thing. It's basically a unique identifier for that particular operation, and Swagger specification say is that you have to...an operation ID has to be unique. We'll come to operation ID later. Then you've got a bunch of parameters. Now parameters like your query parameter over there, you've got parameter called name. But it could have well been a parameter that resides inside a body, for example when you make a post request. And that's reflected on the fact that the parameter is called name, name is name, confusingly, and it resides in query. And the parameter itself you can actually describe the parameter. It's the name of the person to whom to say, "hello" to. You can specify whether that particular parameter is required or optional. So I've said it's optional, and the type of the parameter itself. Now type itself can be a complex thing. Swagger builds on JSON Schema and JSON Schema itself defines means and mechanism for defining complex types whether it's integers or arrays, and also bounds on those particular types. So you can do things like mean length, max length, exclusive length. You can use, for example, regular expression to define what the format is going to be and things like that. And that's fine, you've got the parameters you know which is name, and then you got a section over there that's called responses. If you were to hit this particular endpoint, hello. These are the possible things that your server will send back to you that is what response is. So I've set 200 which is an HTTP status code, that's how responses are for HTTP. It will be a case of success and what success is, is basically a complex thing as a Schema. Some of you might say that why couldn't I define it in-line? I could have. I just wanted to show you that in Swagger itself because it uses JSON Schema, it is possible to define stuff in-line or out of band, in separate section which we'll see shortly. So it's a Schema and it's got a weird syntax dollar ref that basically tells you that in the same Swagger document somewhere, there is a section for definitions and there is an object called hello response and that's what you're going to get back. Then there's also another response in there that's called default which is if none of the responses I'm sending back match, this is what you're going to get. So for example when you are writing rest APIs, you usually do things like 200, 400, 5 or 3. Usually if you're lazy, it's either 200 or something really bad happened. So the default is actually an error, and again, I've used the schema to define what that error looks like which you can just see right at the bottom. So in definitions which is YAML section as I've said, the Schema will be found in the definition section. You've got hello response and error response. What is hello response? It's got a property called message that's required and the type of that is a string. Error, basically the same thing. It's got a required property called message and the type is also string. Now one of the thing to notice here is that we could have sent other parameters along with this other payload also because they are not required parameters, but you can always describe them here so that it's easier for people to know that I will always get message but I might also get some additional information with it. In fact, with a valid Swagger parser, you can actually tell that your response request or any of these schema that you're sending out or receiving should always be checked so that even if we accidentally introduce another field, even if you've said that it's not required it will blow out on your face and say that "You know what, you're sending out stuff or you're receiving stuff that you didn't say that you were going to." So basically just the summary of what you basically saw earlier is... This swagger API basically, you provide the description of the API, the pathways available and the type of content it can handle, we have seen that. Defining a set of rest operations and describe what the payload and the response looks like using JSON Schema. And you can optionally define security parameters for all of these operations which is important. We will hopefully see it because when you're writing rest API, it's important that you have security parameters on it. You can obviously have Open APIs, but usually you also have APIs that are secured either by access token or by some form of signature on your header, API key, things like that. So that was Swagger in general. How about Swagger and node, because this is LNUG, and I'm here to talk about node also. The support for all major node frameworks in many different forms and shape. You've got high level libraries that you can find on NPM like Swagger-Node, HAPI-Swagger, Swaggerize-Express, Volos. Looks like everyone has a library that they've written. You've got low level packages like Sway. So if you don't want to deal with really high level stuff, and you're that sort of person who writes everything from scratch, something like Sway will help you because it's got the low level building blocks to parse, Swagger specification, etc, etc. Suddenly, choosing the right framework will actually require some thought. I mean, this is one of things, in general in node ecosystem, that's the case. It's not like Java, if you are doing certain things, you know that I just should be using string or I should be using an IO, blah, blah, blah. There are frameworks that have won, but in node world, winning is not that easy. So you have to take these libraries, look at the used cases you have and figure out whether any or one of these will actually fit your used cases. So things like you should always think about performance because it's a machine readable format using JSON schema. Parsing speed matters. Whether your JSON is being parsed synchronously, asynchronously whether it's streaming, whether you have flexibility. When you're writing a node server you should be able to define middleware and things like that. Also, what features does it provide? Is it compatible with the entirety of the specification or have lost out on a few things and of course traction. That's also very, very important. You don't want to write system critical services and then there's later have a left band in your hand. Yeah, basically do your due diligence. All right. So I'm going to pick a stand alone reader for user Starcount. It's called Swagger Node. That's where you'll find it. It was written by people at Apigee. Apigee has recently been acquired by Google. Apigee was one of the first... one of the very first proponents of API as a service. So these guys provided enterprises with means and mechanism to deploy APIs and hide their legacy systems behind it providing services like caching, analytics, and a bunch of other stuff. So they have a lot of experience with this thing, and they basically donated Swagger Node to the open source community, provides great abstraction out of the box which is good. It's not perfect. There is some learning curve required for advance features, but it's very close enough, and it's also very, very fast from what I've seen. How do you get started with something like node swagger? You install the Swagger package, and once you've installed Swagger, you can just basically create a scaffolding project by just running swagger which itself is a command line tool, and you can say a swagger project create and you can say a swagger project start, and that's about it. It will create a default project for you with a folder structure that's on the screen over there, and what it will give you is your standard express base one. While you do create your project, you do get it a choice whether you want to use Express or connect or HAPI or RESTify. I've chosen Express for this example. You get your standard [inaudible] very straightforward stuff. It generates controllers for you, it generates the test for them, generates the swagger file as an example for you which is very much like the hello one that we saw. And, yeah. I'm going to give you a quick peek. And I basically cheated and created the hello project for you. So if I do LS over here. If I do something like that you can basically see that's the folder structure I received. Now, one of the things I said in the presentation was swagger project start. So if do that... Sorry. Yeah. Better? Yeah? Swagger is a common line tool. Now, ideally, when you are writing node obviously on production, you want to start your node server with something as simple as node.js, but when you're developing, Swagger itself gives you a nice wrap around on top of node one. So it watches your whole file system blah, blah, blah. If you make any changes your Swagger file or your code, it will automatically restart. So when I do this, it says that it will start the server, and you can basically see that it says, "try this." Hit that end point curl it. Too lazy to curl so I've got postman over here. Now, if I hit that, do a send, response, "Hello, T." And if I say, "Blah, blah, blah." That. Now going back to the project itself. Now what happens if I say over here one of the parameter is actually required false. So if I do require true and go back here, as you can see, the server has restarted because I've made a change which is good, and instead of doing that, I just get rid of that and send it. Immediately, you get an error message back because you haven't kept to the interface that the API expects. Now, if you do go back to the code and if you go back to the actual app.js, as you can see it's actually very simple. All we had to do was include a express middleware, instantiate our express app the standard stuff and basically register that with swagger express which basically takes care of reading the YAML file, setting up the roots, and setting up all the necessary plumbing that it requires. Funny thing, hello.js which is our controller and this where things start to fit together. Remember, we said that in our Swagger path section in the get that there is something called an operation ID, that's called hello over there, and hello.js explores the function called hello. So that's how operation ID and your actual function call reconcile. All right, so what happens with this hello function? It's called your classic express signature req, res, next. Request, response and your next call to whatever middlewares that's handling next. Now look at this, if you were writing a standard node or express style app, at this point you'd probably read req.params or req.query to retrieve the value because swagger actually injects its owns parsed values as it sets up its plumbing under the hood, you get that from req.swagger.params. So because our query parameter was called name, if I just said name, if I just said req.swagger.params.name, it would actually give me the swagger object that defines name itself, but when I do .value I actually got the value. Now one really nice side effect of this is because swagger is a machine readable specification at this point in time, when I've got req.swagger it's all been handled for me. Everything has been parsed, everything has been sanitized, if it wasn't, swagger would have sent the 400 back to the client. So I don't actually have to do things like parsing of the data, assembling it, sanity checking it, checking the bounds, blah, blah, blah. I can just use JSON Schema to define this is exactly what the data should look like as an input and Swagger will basically take care of validating it and sending it. So you save a lot of [inaudible] by just doing this. And what am I doing over here? I'm just basically assembling util.formatting it and sending it back as res.json. Now there is one thing. I don't know if anyone has actually noticed. We said that in the swagger EMO file, the response is actually supposed to be an object if you can see it with the property message. Now over here, I'm just sending the message as a string. So how does this work? Actually, it doesn't. Swagger also provides output validation which is turned off by default, and the reason why they've turned off output validation by default is because when you're writing production and creates stuff, validating your output can be expensive. What you really want to do is turn on output validation when you're developing, when you're testing, and in production basically be able to turn that off. Good thing about output validation is that everyone who usually writes a node server or REST API is always concerned about input validation. A lot of people don't actually think about output validation but output validation is also important because you've got specification and you've hand it over to some other guy they're writing code against that specification, if you break that specification you should know about it. So how do you do that? Well, if you go back to app.js over there, there's this tiny little piece of code that I have commented in there, what it basically said is that if it wasn't production then basically we could listen on a response validation error message from the Swagger framework itself, and what it would do is that if you are doing anything dodgy in terms of responding it would basically tell us. So if I do that, go back... Oops, sorry. Cool, cool, look. Because it should be started, give me the syntax error anyway. Sso if I try doing that, of course, this wouldn't parse to start with because there's name equals... If I do something like that, and if I do it. Oops. It still ran, but guess what? Ideally, when you're testing you probably want to do other process.exit or have anything sensible in place that enables you to basically cache this because no one is going to sit in your continuous integration system to figure out what error happened, but this is what you'd normally get back from it when you have response validation [inaudible], it will basically tell you've done something wrong, go and fix it. So I'm gonna turn that off. Now going back here. So we have seen the fact that swagger has taken care of sanitizing your parameters. It's giving you the opportunity to concentrate on logic which is always nice as oppose to handling lots of [inaudible] stuff. And we did say that we were going to talk a little bit about securing stuff. So for example, you've got the hello API over here, and it's a precious API, you don't want any [inaudible] and harry hitting it, and getting a nice hello from precious server. So for example you decide, now you want to secure it somehow. So what are the standard security measures that are popular nowadays? You can have have OAuth two, which you can send a bearer token in the header, authorization header. You can use basic OAuth not so cool anymore but still works. Or you can use API key which you can put either in the request header or you can put into the request as a parameter itself. So putting API keys in parameters, query parameters, isn't generally recommended simply because if you're logging all that stuff in your server somewhere you're also logging other people's random API keys, not a good thing to do. So usually people stick that stuff into the header. So I've done the same thing. So to start with that, what you do is that, rather in swagger, you can say that you have security definitions. I don't know if you guys can actually see it, but right at the bottom if I make it even bigger and security definitions is just another section just like definitions. You can put multiple security definitions in there, and use any of those if you want. I'll just one. I've called the security definition api key. The type is API key. The type is important because the swagger understands a couple of these things. It understands API key, it understands bearer token, and it understands basic OAuth. I've said that expect an API key in header. When I say in header that means in the header of the HTTP request and the name of that header is going to be api_key. All right, so that's the definition. Now, if I want to secure my hello API, I can go in there and basically say... Nice, disaster avoided. All right, so if you go back and try doing the same operation now because we have to put a security key in there. Oh, unauthorized. How did that happen? And how can we get around it? Well, when you do...when you use swagger node it gives an opportunity to basically write so-called fittings which are basically like middleware that you can plug into your project. I've created such a middleware and it's called a security handler, and the way these things are introduced is when the project itself is generated with the Swagger EMO file, you also get a default EMO file which is basically like configuration for the underlying Swagger Node module itself injected into your project and you can basically inject different middleware for your parts, and define how validation is going to work, how error handling is going to work. So this is how it looks likes. It uses a mechanism called bagpipes which is actually pretty sophisticated but out of scope for this particular discussion, but the idea is that what I did is that, in the Swagger security section I basically said that all the security handlers will be found in a file called API Swagger security handlers. So it basically has the mechanism to poke into that particular folder, read that file, read the exported methods and invoke them whenever it hits a secured API endpoint. So what does that particular file look like? It looks like this. So it's got a function called API key, and it explores API key. That's important. Remember in the swagger file itself, we actually named it API Key with capitalization, and what it does is it gives you a standard express style middleware function again, req, res, next. Everything that you are familiar with. And what I said over there is "If req.headers has api_key," which is what we said, is what it's going to look like, If it's equals to some random string, then return next, no error. Otherwise, just return 401, and that's exactly what happened. We got a 401 back. So if you change it back to something like that. If you take that particular secret, we go there, and we go into the headers. So for example, let's say I have node, so if I do api_key and do that and take that request, stick it in there, and send it. How do I unlock because it's actually had the security middleware has done its validation and it said you can go ahead and return this value. So that's basically swagger in action, but when you use something like swagger node you also get a little bit of a bonus. So I'll go back to my screen. This is where my original code was. And if I just navigate to... If I do something like swagger edit, what I actually end up is an editor for the specification itself, and as you can see, it's rather specification, it even knows what the security is going to be like. So there you can stick that in. This is the interface that will be familiar to a lot of people who have seen this on third party websites where they let you drive their API and things like that. Now you know where it actually comes from. So you've got all the parameters and their descriptions, the models themselves documented. If you want you can always try the operation out also. So for example over here, if I say name is LNUG. I can send a request. They are unauthorized because I haven't authenticated the API. So if I go back, and oops, basically take that, stick it in there, authenticate, you've got a green check mark next to it, and I do a send request. Yeah, port is different so. What? Good tip. So I press that even though the port didn't match. It still worked. Maybe something with local host, but yeah, I mean, the nice thing is that you can basically change your specification and try it out as you write it and test it. So again, node also gives you the ability to write mocks which is bit out of the context of this particular discussion, but you can always look it up and you can try it out. Yeah, there are a couple of other websites, swagger.io. It's got this basic [inaudible] the library we talked about, and there's a dude called API handyman. I don't know what his name is but he's really good. He writes a lot about API and specifications, and design patterns and things like that. Highly recommended, check it out.