Hi @stefanalund , we already have this feature implemented for OpenAPI, although I'm a bit confused how this could be implemented into Backstage.

Right now we are repurposing our backend to be feeding a Backstage frontend, and we'll have this feature straight away, but it would require our service discovery, and basically our backend to be running. We could definitely separate it into its own backend, and easily provide an API overview page, but then I'm not sure how we could also integrate it to our service page.

As far as I recall, you mentioned that in that case the information passing from one backend to other would be taken care in a separate layer, but it's still a bit blurry, could you explain a bit further?

I can see this working if we'd have a OpenAPI plugin for example, and then a dependency in our Services plugin, and only in that case we'd display the specific OpenAPI definitions of a service in their detail page, is that how it's supposed to work? Or is there a better approach?

Here is how it looks in our tool

@aexvir it is fussy to me as well :) When we get further on the Backstage backend architecture, I think it will be easier to see the path forward. Ping @freben @mbruggmann

In our landscape at bol.com we have a setup whereby every service has its own "internal" page that hosts, among other things, multiple Swagger UI pages (one for every "API Group", which can be something like the "v1 api", the "v2 api", the "admin api", etc). These all come from separate OpenAPI spec files that are also hosted by the application.

This data is something that could be directly consumed by a Backstage plugin, removing the need to host the Swagger UI in the application itself.

Speaking more broadly however, we've prototyped an API Registry to address a number of more general concerns. Note, the registry is definitely not production ready, but maybe I can explain in a some long-winded detail (sorry) what the problems are that it was trying to address. Hopefully it can inspire some of the design of Backstage's API model.

Queryable API specs

One of the areas we want to invest in is client generation for APIs. In the OpenAPI space there's tooling like openapi-generator, there's similar tooling for gRPC. This can be then integrated in the build pipeline of a consumer so that they can use your team's API through a generated client, avoiding any unfortunate API mismatches.

The catch, however, is that you need to have the OpenAPI/gRPC spec file in order to generate anything. So the process of downloading an API spec file should be automated somehow, e.g. through a build plugin (like a Maven or Gradle plugin in the Java ecosystem). It would be really useful if this plugin could point to a centralized API and just say: Give me the latest version of the API for service foo in the 2.1.x range, similar to how you'd set up a Maven or NPM dependency. The precise version to download can then be resolved either server-side or in the plugin, depending on how this download API is designed.

Notifications of changes

Just like it is useful for teams to get notifications about deployments, infrastructure changes or security breaches, it would be very nice to be able to tell the consumers of your API that a new version is available. This could be automated by having some kind of 'watch' functionality on the API page of a service. As a consumer I could register my interest and get notified and, conversely, as the owner of the service I'd have an up-to-date list of who's interested in my API.

Quality checks

As the owner of an API, it would be nice to integrate linting tools like Spectral to show me the quality level of my APIs.

Additionally, other tools could be added that show if minor API releases are backwards incompatible or not, etc.

Proposal

So as I mentioned a number of these features are already part of Katalog, but it makes much more sense to me to make it less monolithic and simply combine a number of separate tools via a Backstage plugin. Katalog could be used as the bare API registry, in that case. But apart from that I also hope all of this makes some sense and can form as an inspiration for further discussion here :)

This is super interesting! All of what you say sounds familiar and are problems that fit well under the Backstage umbrella. I won't be able to write up a response right now, but I'll get back to it as soon as possible!

As the owner of an API, it would be nice to integrate linting tools like Spectral to show me the quality level of my APIs.

I really like this. Could see it as something that you would run in the background and show any warnings/errors in the Backstage plugin. This is a common pattern for Backstage at Spotify; we use visualisations in Backstage as a way to inform and nudge engineers towards following best practices.

It's one of the things I implemented in our API Registry, but it led to all sorts of unwanted complexity (embedding a Node application like Spectral in a Kotlin app, etc). A Backstage plugin would be much better suited for this. Frontend editing components like CodeMirror even have a very straightforward way of visualizing the linting results.

Interesting, I think there are a few aspects to unpack here:

• Some type of registry or inventory of which APIs exist, their type (OpenAPI, GraphQL, gRPC), the component/system/repo they belong to, and where to query their schema. This could possibly live with Backstage?
• Indexing the actual API schemas, either from source code, through reflection at runtime, or through a manual process. Could be specific to the type of APIs and to whatever process is used by the company. This is where I see tools such as Katalog coming in.
• UX for Searching, Browsing and Visualizing APIs. Probably a good fit or Backstage

Yes, that seems about right. Katalog could then be reduced in scope quite a bit to 'just' host the API specs and providing a flexible query API for retrieving them.

One thing that would have to be carefully considered is how the link is made with the metadata that is stored in Backstage: Is the service name in Backstage a unique key under which Katalog can then store API specification files? How does it work when a service is deleted, etc?

In Zalando, we implemented this feature in a standalone API Portal supported by the API Repository providing a historic perspective on all deployed APIs. The API repository is supporting an easy search language that directly provides results from an ElasticSearch engine.

Content is pushed to the API Repository via a job running every 5 minutes to collect all current deployments and resolve their APIs from two potential source:

1. The Docker image via a service called API Scanner which is working as a big cache, or
2. as fallback by querying the .well-known/schema-discovery endpoint implemented by the application.

We prefer the first source, as it provides a more precise picture in case of multiple application stacks, however, we have gateways publishing their dynamic target APIs through the legacy schema-discovery. We actually would prefer them to push this information directly to the API repository, but we have not provided the API for this yet.

We although have some quality metrics for APIs based on our API review database as well as our API linter Zally that we present in the API Portal.

In the API Portal we are today a bit stuck from technology perspective. Adapting the Swagger library was challenging 3 years ago and we introduced some dependencies that are not maintained any longer and hard to substitute. A main aspect is actually connecting the API with all provided context information as you can see in the picture. I can envision that this will be the biggest challenge in Backstage as this information needs to be provided by other plugins.

However, the backend is very simple and robust. May be we can contribute with our experience here and even provide some open source components.

@tkrop thanks for the write-up!

Is the API Portal (and perhaps API Scanner) open source or something you would consider releasing as part of Backstage? Sounds like it could be a good match.

I imagine that API scanning and ingestion step may be very dependent on a company's underlying infrastructure, thus making it hard to provide an end-2-end solution. Backstage could provide a standardised UI (through a plugin) and a backend with a clear ingestion API.

@stefanalund I think we are willing to contribute, if the community is interest enough.

Interestingly, the API scanner is the part to open source easiest while, we think, already having a very high value for the community. It has nearly no company specific hooks, and the existing ones can be removed or disabled easily. It has only very small technical debts and provides a good potential for extension by the community, e.g. by supporting other API specification and documentation formats. A simple backstage plugin could already integrate against the API scanner API to visualize APIs based on Docker images names.

Next easiest would be to opensource the API repository consisting of a Postgres and ElasticSearch data storage for providing the historical API information. With relatively small effort, we could remove the legacy API and integrate the search feature from the API Portal to make it usable for the community. While the data model is sufficiently generic it is a still a bit incomplete, since we have not developed the whole envisioned model. Also the API has some rough edges and supports only a small number of use cases yet. Anyhow, nothing that cannot be fixed by the community.

To make this work, users need to implement a discovery job/service, that populates the API Repository using the data from the API Scanner. This is the company specific part of our architecture that also offers the freedom of adding APIs not detectable via the API Scanner. Finally, a backstage plugin needs to integrate against the API Repository search API to search and present the historic API data.

@tkrop It would be great if the scanner could be optional. At bol.com we are aiming for build-time linting and registration of APIs (i.e. before the service goes live, to prevent any accidents). For JVM services we could then provide a Maven plugin that performs the scan in the local project before uploading to your API repository.

Second, perhaps it would be nice if the Elasticsearch dependency is optional and it could be made to work just with the Postgres full-text search functionality. ES is quite heavyweight and it would increase some operational burden. This can also be a community effort of course, though :)

@tkrop I'm know for a fact that there are several companies that would be interested in this. Also, speaking on behalf Spotify, we don't have a good story for API documentation and would gladly adopt something like what you are proposing 👍 Ping @garyniemen

@RoyJacobs yes, build time linting is supported by zally.

@tkrop Ah, great! We'll finally start a pilot project with Backstage soon and that will definitely be on my list, then :)

@freben @Rugvip @mbruggmann now that the service description file format is defined, could you provide an example (in this issue) for how you would link up a OpenAPI spec yaml file with the component?

For everyone else in this thread: would you be willing to take on building this plugin? It should be visible on the service (component) page and could use the open source Swagger UI (as demonstrated by @aexvir above):

We are starting work to create an end-2-end demo of Backstage with a few key plugins, and I think this one should be part of it.

@freben @mbruggmann, what do you think about starting out simple with backstage.io/api-url + backstage.io/api-type annotations or something similar? I'm thinking local API descriptions might make sense for individual services anyway, even with a separate Api kind that are attached to systems

@Rugvip Yeah, starting with just having the type and url sounds good. Though I would probably have gone with a separate kind, and then link them to components through the name. Eg this:

---
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: frobs-awesome
spec:
  type: service
  implements-apis: # <-- not sure how to call this, nor if it should be an annotation/label instead
    - frobs-awesome-api
---
apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: frobs-awesome-api
spec:
  type: openapi
  url: https://...

Mostly because I consider the Component to API relationship many-to-many.

@mbruggmann Yep, agree that we should go with the from the beginning if it doesn't snowball into a bunch of other work and problem solving.

The only remaining question then is, how does this Api kind entity enter the catalog? It would probably not be convenient to have to specify it in its entirety, if the ingestor can find files that are next to the entity file (or pointed to, by relative path, by the entity file).