Using The OIDF Conformance Suite To Help You Implement an Openid Provider

The OpenID Foundation, or OIDF for short, is a non-profit which sticks its fingers in a number of identity-related pies. Of particular interest is openid-connect, a protocol built on top of Oauth2, to provide an identity layer. Of ever more interest is the OIDF’s conformance suite, a web application which allows users to run extensive suites of tests against their own implementations of the OpenID Connect protocol, to self-certify that they are compliant with the spec.

I’m particularly interested in it, as my day job for the past few years has been to maintain a fork of this code, and extend it, umm, extensively, to test not only the OIDC protocol, but also the functional APIs being protected by it in various Open Data initiatives. The first of which was Open Banking Brazil. Along the way, I’ve contributed code back into the OIDF codebase.

However, my current interest is in the OIDF flavour of the conformance suite. Why? Because, mostly for fun, I’ve been writing an OIDC provider library in Java, and a Spring Boot implementation of the protocol, and am using the OIDF suite to test it. Yep. I am indeed a riot at parties.

I figured I’d document my journey through building this library, and explain how I’m using the conformance suite as an aid.

The OIDF documents in detail how to run this locally, so I won’t duplicate their efforts here. But once you’re up and running, you can visit https://localhost.emobix.co.uk:8443 and see the suite in action. In dev mode, there’s no login required. You should see something like this:

Ok Let’s Run Some Tests

From here on in, I’ll be assuming you have some sort of OIDC provider running somewhere, accessible from your local machine. I’ll also assume you’ve implemented OIDC discovery. That is, your implementation responds to /.well-known/openid-configuration with something resembling sane results. You can also configure the tests with metadata manually, but eh, discovery is the way to go.

Ok, click “Create a new test plan” and you’ll be given a form, with a drop-down at the top where you can select a test plan. Choose the one called “OpenID Connect Core: Basic Certification Profile Authorization server test”, select “discovery” for server metadata and static_client for client registration.

The rest of the form is straightforward enough:

• alias - this is a slug that will make your test vaguely unique amongst them all. For running locally, “test” is fine.
• Description - a vague description of what you’re testing. “Testing my OIDC provider” is fine.
• Publish - no
• discoveryUrl - this is the well-known of your provider

It also asks you to configure up to three oauth clients. The first two can in fact be the same client if you wish, as long as it can use both basic auth and client secret post authentication methods. The third is for testing client takeover.

Note that these clients will need to be configured with a redirect_uri of https://localhost.emobix.co.uk:8443/test/a/mytest/callback, where ‘mytest’ should be replaced with the alias of your test.

Click ok, and we’re ready to run a test. You will see a number of scenarios, each with a ‘Run Test’ button next to them. Simply run these, one at a time, and watch the logs. You will at some point see a button requesting browser interaction. This is typically where you will log in to your OP and grant some permissions to the conformance suite. This is scriptable by the way, but I’ll leave that for another post.

If everything goes green on one test, you’re one step closer to OIDC conformance! What we’re interested in, though, is the failures. Intuitively, failures come up red in the logs, and are generally very descriptive of what went wrong. Here’s one of mine

This tells me a number of things. I can see that the test requires my OP to support client secret post authentication, and that it doesn’t. It tells me what it expected to see in the discovery document, and what it actually saw. It also tells me, if you’re interested, the class in the conformance suite which found the error. In this case, EnsureServerConfigurationSupportsClientSecretBasic. There will be a Java class in the conformance suite code called this, which will contain the logic for this step.

Ok, that’s a fairly easy one to fix. Let’s look at something more tricky.

This is where the conformance suite starts to be a bit more useful. It’s told me that there is a claim missing from the id token generated by my OP, and why it was expected. But it also has links to OIDC specifications which I’m failing to meet, or which are relevant. These failures are where you can really start to drill into specifications and understand what you’re doing wrong.

Some tests, you cannot actually pass.

This particular test, oidcc-prompt-login, uses an authorization request prompt to advise my OP to ask the user to re-authenticate. As this all takes place in a browser session with my OP, the conformance suite can’t reasonably examine it, and so instead asks you to take a screenshot of the login page and upload that to the test. Should you actually submit your certification results to OIDF, this will be reviewed.

And that’s it. That’s the basics of the process. I implemented the provider from memory, and from reading specs, as much as I could, then ran it locally and started firing these tests at it, over and over again, fixing issues, adjusting and refactoring code to meet specs as I discovered failures. When your OP has passed all of these tests, you’ll have some lovely green blobs to marvel at:

In another post, I’ll dive a bit further into what some of the tests are doing, and how they relate to various specs and what an implementor might do with them.