Using ADFS with internal CA certificate for OIDC

I am attempting to set up OIDC to an internal instance of AD FS that uses an internally trusted root CA

Since the Firezone VM is ‘internal’ in my network, it attempts to communicate with the AD FS server itself (instead of the AD FS WAP). The internal AD FS server uses a TLS certificate issued from my AD’s internal Enterprise Root CA (via in internal Intermediate CA)

While the underlying OS (Debian) trusts the Enterprise CA (via the Root CA PEM being included in the OS’s /usr/local/share/ca-certificates/ directory), I can’t yet figure out a way to get fz_http to pick up and trust certificates that chain to my Enterprise Root CA.

I did successfully get the web interface to use a TLS cert chained to my Enterprise CA by pointing default['firezone']['ssl']['certificate'] = to a .pem (containing only that leaf cert). I didn’t expected this to be an problem since clients accessing the Firezone’s web interface will already trust certificates that chain to the Enterprise Root CA but from past experience sometimes other components use this cert and chain for other purposes.

I also manually modified /var/opt/firezone/ssl/cacert.pem to include my internal Root CA and also tried to configure default['firezone']['database']['ssl_opts'] = {cacerts:"/var/opt/firezone/ssl/cacert.pem",}, which didn’t change anything. (firezone-ctl reconfigure was run between all tests.)

Is there a place I am missing to provide the internal Root CA (or to correctly append the cacert.pem file)?

For completeness, the firezone.rb config entry :

default['firezone']['authentication']['oidc'] = {
  adfs: {
    discovery_document_uri: "",
    client_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    client_secret: "xxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxx",
    redirect_uri: "https://vpn.external.domain/auth/oidc/adfs/callback/",
    response_type: "code",
    scope: "openid email profile",
    label: "ADFS"

and the error generated at /var/log/firezone/phoenix/current

2022-09-26_21:16:00.68838 17:16:00.684 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0 at (http)
2022-09-26_21:16:00.69071 17:16:00.690 [info] Access FzHttpWeb.Endpoint at https://vpn.external.domain
2022-09-26_21:16:00.84346 17:16:00.819 [notice] TLS :client: In state :certify at ssl_handshake.erl:2098 generated CLIENT ALERT: Fatal - Unknown CA
2022-09-26_21:16:00.89363 17:16:00.858 [notice] Application fz_http exited: FzHttp.Application.start(:normal, []) returned an error: shutdown: failed to start child: FzHttp.OIDC.StartProxy
2022-09-26_21:16:00.89366     ** (EXIT) an exception was raised:
2022-09-26_21:16:00.89366         ** (MatchError) no match of right hand side value: {:error, :update_documents, %HTTPoison.Error{reason: {:tls_alert, {:unknown_ca, 'TLS client: In state certify at ssl_handshake.erl:2098 generated CLIENT ALERT: Fatal - Unknown CA\n'}}, id: nil}}
2022-09-26_21:16:00.89367             (openid_connect 0.2.2) lib/openid_connect/worker.ex:55: OpenIDConnect.Worker.update_documents/2
2022-09-26_21:16:00.89368             (openid_connect 0.2.2) lib/openid_connect/worker.ex:23: anonymous fn/1 in OpenIDConnect.Worker.init/1
2022-09-26_21:16:00.89369             (elixir 1.14.0) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
2022-09-26_21:16:00.89370             (elixir 1.14.0) lib/enum.ex:1552: Enum.into/3
2022-09-26_21:16:00.89371             (openid_connect 0.2.2) lib/openid_connect/worker.ex:22: OpenIDConnect.Worker.init/1
2022-09-26_21:16:00.89371             (stdlib 4.0.1) gen_server.erl:848: :gen_server.init_it/2
2022-09-26_21:16:00.89372             (stdlib 4.0.1) gen_server.erl:811: :gen_server.init_it/6
2022-09-26_21:16:00.89375             (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
2022-09-26_21:16:02.42360 {"Kernel pid terminated",application_controller,"{application_start_failure,fz_http,{{shutdown,{failed_to_start_child,'Elixir.FzHttp.OIDC.StartProxy',{{badmatch,{error,update_documents,#{'__exception__' => true,'__struct__' => 'Elixir.HTTPoison.Error',id => nil,reason => {tls_alert,{unknown_ca,\"TLS client: In state certify at ssl_handshake.erl:2098 generated CLIENT ALERT: Fatal - Unknown CA\n\"}}}}},[{'Elixir.OpenIDConnect.Worker',update_documents,2,[{file,\"lib/openid_connect/worker.ex\"},{line,55}]},{'Elixir.OpenIDConnect.Worker','-init/1-fun-0-',1,[{file,\"lib/openid_connect/worker.ex\"},{line,23}]},{'Elixir.Enum','-map/2-lists^map/1-0-',2,[{file,\"lib/enum.ex\"},{line,1658}]},{'Elixir.Enum',into,3,[{file,\"lib/enum.ex\"},{line,1552}]},{'Elixir.OpenIDConnect.Worker',init,1,[{file,\"lib/openid_connect/worker.ex\"},{line,22}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,848}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}},{'Elixir.FzHttp.Application',start,[normal,[]]}}}"}
2022-09-26_21:16:02.42367 Kernel pid terminated (application_controller) ({application_start_failure,fz_http,{{shutdown,{failed_to_start_child,'Elixir.FzHttp.OIDC.StartProxy',{{badmatch,{error,update_documents,#{'__exception__' => true,'__struct__' => 'Elixir.HTTPoison.Error',id => nil,reason => {tls_alert,{unknown_ca,"TLS client: In state certify at ssl_handshake.erl:2098 generated CLIENT ALERT: Fatal - Unknown CA\n"}}}}},[{'Elixir.OpenIDConnect.Worker',update_documents,2,[{file,"lib/openid_connect/worker.ex"},{line,55}]},{'Elixir.OpenIDConnect.Worker','-init/1-fun-0-',1,[{file,"lib/openid_connect/worker.ex"},{line,23}]},{'Elixir.Enum','-map/2-lists^map/1-0-',2,[{file,"lib/enum.ex"},{line,1658}]},{'Elixir.Enum',into,3,[{file,"lib/enum.ex"},{line,1552}]},{'Elixir.OpenIDConnect.Worker',init,1,[{file,"lib/openid_connect/worker.ex"},{line,22}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,848}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{'Elixi
2022-09-26_21:16:02.42750 Crash dump is being written to: erl_crash.dump...done
2022-09-26_21:16:02.91130 Firezone detected a service crash loop. Taking service down. For support please email and include a copy of these crash logs.


Hey @hmhackmaster – the issue you’re hitting is the Firezone installation’s OpenSSL lib is failing to verify using its local cert store.

When Firezone boots up, it retrieves the jwks config from the jwks_uri present in the openid configuration fetched from your discovery_document_uri. It’s failing at that step due to the above issue.

Try adding your CA to /opt/firezone/embedded/ssl/certs/cacert.pem and see if that helps.

Yep, makes sense, that’s exactly what I expected.
I just checked that location and it looks like /opt/firezone/embedded/ssl/certs/cacert.pem already matches the /var/opt/firezone/ssl/cacert.pem file that I manually edited, so somehow the file got updated automatically.

I did a pretty haphazard job at adding my cert to the cacert.pem file, so let me try to clean that up first.
If that is where my Enterprise CA is expected to be, I’ll go ahead and use the base OS’s cert store (which already includes my Enterprise CA) to automatically generate it (especially since I don’t know how often that file gets updated and the cause of the breakage isn’t immediately obvious).

I am looking forward to getting this going! My lab environment hasn’t had a good VPN solution for quite a while so I am very excited!

No luck thus far.
Steps I have completed:

  • Confirmed that my AD FS’s web server provides the Leaf and Intermediate certs, so only having the Root CA in the cacert.pem should work fine.
  • Went ahead and added my Intermediate CA cert to the cacert.pem just in case. Service still failed.
  • Used the OS’s hosts file to override the to resolve the IP of the AD FS WAP (which has a publicly trusted TLS Cert) and made necessary firewall policy adjustments. Confirmed the service started properly (login attempt gave me an “Internal Server Error” but that’s another problem, I am sure).
  • Restored the hosts file and used openssl to see if the cacert.pem would work correctly:

The original cacert.pem correctly errored out:

openssl s_client -verify_return_error -connect -CAfile cacert-orig.pem
verify error:num=20:unable to get local issuer certificate
139654557963584:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1914:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 3276 bytes and written 311 bytes
Verification error: unable to get local issuer certificate

My *modified * cacert.pem validated correctly:
(note that the AD FS TLS cert has it’s Subject/CN set to something different and includes a SAN for the FQDN being used; I assume that is not an issue but I figured it’s worth mentioning.)

openssl s_client -verify_return_error -connect -CAfile /var/opt/firezone/ssl/cacert.pem
verify return:1
verify return:1
depth=0 CN =
verify return:1
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-384, 384 bits
SSL handshake has read 3327 bytes and written 462 bytes
Verification: OK

Please let me know if I am missing anything so far; I’ll continue to poke around in the meanwhile!

@hmhackmaster Hm. It looks like we may need to pass the custom cert to the underlaying HTTP library Firezone is using for the SSL connection.

See openid_connect/openid_connect.ex at master · DockYard/openid_connect · GitHub

I’ve opened Expose SSL_CLIENT_OPTS variable for client side SSL connections · Issue #996 · firezone/firezone · GitHub to track. This would take a new config option default['firezone']['ssl']['client_opts'] as a hash of options to pass to the Erlang SSL module.