Firezone docker container crash

I have a strange problem with the fact that the container firezone started crashing after startup. Here is my compose.yaml and the lines I found in the docker log. I tried to find out what the cause is, but I did not find similar problems here or anywhere else. My host system - VPS with Centos 9 with latest updates.

UPD:
This way the configuration worked for about a week, without making any changes. Users registered to the site and there were no problems.

x-deploy: &default-deploy
  restart_policy:
    condition: on-failure
    delay: 5s
    max_attempts: 3
    window: 120s
  update_config:
    order: start-first

version: '3.7'

services:
  caddy:
    image: caddy:latest
    volumes:
      - /data/firezone/caddy:/data/caddy
    ports:
      - 80:80
      - 443:443
    command: caddy reverse-proxy --to firezone:13000 --from ${EXTERNAL_URL?err}
    deploy:
      <<: *default-deploy

  firezone:
    image: firezone/firezone:latest
    ports:
      - 51820:51820/udp
    env_file:
      # This should contain a list of env vars for configuring Firezone.
      # See https://docs.firezone.dev/reference/env-vars for more info.
      - .env
    volumes:
      # IMPORTANT: Persists WireGuard private key and other data. If
      # /var/firezone/private_key exists when Firezone starts, it is
      # used as the WireGuard private. Otherwise, one is generated.
      - /data/firezone/firezone:/var/firezone
    cap_add:
      # Needed for WireGuard and firewall support.
      - NET_ADMIN
      - SYS_MODULE
    sysctls:
      # Needed for masquerading and NAT.
      - net.ipv6.conf.all.disable_ipv6=0
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.forwarding=1
    depends_on:
      - postgres
    deploy:
      <<: *default-deploy
  postgres:
    image: postgres:latest
    volumes:
      - /data/firezone/postgres:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: ${DATABASE_NAME:-firezone}
      POSTGRES_USER: ${DATABASE_USER:-postgres}
      POSTGRES_PASSWORD: ${DATABASE_PASSWORD:?err}
    deploy:
      <<: *default-deploy
      update_config:
        order: stop-first
12:42:00.125 [info] Migrations already up
12:42:03.244 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0 at 0.0.0.0:13000 (http)
12:42:03.248 [info] Access FzHttpWeb.Endpoint at https://nfox.net.ru
12:42:03.626 [notice] Application fz_http exited: FzHttp.Application.start(:normal, []) returned an error: shutdown: failed to start child: FzHttp.OIDC.StartProxy
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, :update_documents, %HTTPoison.Response{status_code: 403, body: "<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 403 (Forbidden)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  <p><b>403.</b> <ins>That’s an error.</ins>\n  <p>Your client does not have permission to get URL <code>/oauth2/v3/certs</code> from this server.  <ins>That’s all we know.</ins>\n", headers: [{"Content-Type", "text/html; charset=UTF-8"}, {"Referrer-Policy", "no-referrer"}, {"Content-Length", "1594"}, {"Date", "Sat, 22 Oct 2022 12:42:03 GMT"}, {"Alt-Svc", "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""}], request_url: "https://www.googleapis.com/oauth2/v3/certs", request: %HTTPoison.Request{method: :get, url: "https://www.googleapis.com/oauth2/v3/certs", headers: [], body: "", params: %{}, options: []}}}
            (openid_connect 0.2.2) lib/openid_connect/worker.ex:55: OpenIDConnect.Worker.update_documents/2
            (openid_connect 0.2.2) lib/openid_connect/worker.ex:23: anonymous fn/1 in OpenIDConnect.Worker.init/1
            (elixir 1.14.1) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
            (elixir 1.14.1) lib/enum.ex:1552: Enum.into/3
            (openid_connect 0.2.2) lib/openid_connect/worker.ex:22: OpenIDConnect.Worker.init/1
            (stdlib 4.1.1) gen_server.erl:851: :gen_server.init_it/2
            (stdlib 4.1.1) gen_server.erl:814: :gen_server.init_it/6
            (stdlib 4.1.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
{"Kernel pid terminated",application_controller,"{application_start_failure,fz_http,{{shutdown,{failed_to_start_child,'Elixir.FzHttp.OIDC.StartProxy',{{badmatch,{error,update_documents,#{'__struct__' => 'Elixir.HTTPoison.Response'

Thanks for the report! This crashing is coming from the OIDC worker – it’s getting a 403 when trying to fetch the JWKS doc from your discovery_document_uri – make sure that’s correct and you’ve set up your OIDC app correctly in Google.

We have improved form validation in 0.6.0 and above if you enter your OIDC config a /settings/security in the portal. That should catch errors like this before they make it to the DB.

This was my first time setting up oauth, and frankly I’m not 100% sure it’s right. I followed the instructions on your site and everything worked. If you mean that google is not getting the page from my site, I think the problem is that the container has already dropped by this time. I tried to move all the files from /data to another location so the containers created all over again in a clean configuration, but the effect is the same.

BTW, first 15-20 seconds site is working, but only a main page and site.com/auth/identity. Possibly it helps to figure out.

The error from your log indicates that Firezone is not able to fetch the OIDC discovery document. Due to the way OIDC works, the IdP (Google in your case) won’t be fetching anything from Firezone. I’d recommend clearing your OIDC configuration using this guide and entering the details again in the web form at /settings/security:

Thanks for response.
I found out what caused the problem. My IP, for some unknown reason, was blocked by Google and Firezone could not access https://accounts.google.com/.well-known/openid-configuration. It’s strange, but maybe you should provide for this in the Firezone code, so that the server does not crash when there is no connection to google and other providers, and we can easily reconfigure server from the UI.

Hey @NFox – we’ve improved OIDC error handling in 0.6.0 and higher if you use the UI form for entering OIDC information. Firezone first validates it receives a valid response from the IdP before saving the config to the DB.

Hey @jamil, I think that’s still a problem. My docker container was crashing endlessly and I couldn’t change the configuration. I had to configure everything from the beginning, and use Auth0, and then everything started working correctly. It’s not a big problem to configure everything from the beginning, but my users had to register again, and I took the time to find a problem that wasn’t obvious to me as a beginner. It might be worth paying attention to this functionality again.