Hi
Question. I know the lowest level of AWS lightsail does not meet the recommended requirements but are the some other requirements for firezone? When trying to deploy (using the $3.5 instance) using the automated installer it will always fail with Error executing action run on resource 'execute[create_admin]'
Deploying on the next level up ($5 per/month) works fine. I am thinking maybe it has something to with disk space?
Just interested to know.
Thanks
Ryan
There shouldn’t be any hard requirements. Could you post more of the error you’re getting? What are the specs of the lightsail instance?
Sure. The specs are Ubunutu 20.04 LTS, 512 MB Mem, 1 vCPU, 20 GB SSD
This is the output of the file /var/opt/firezone/cache/cache/chef-stacktrace.out
Generated at 2022-07-20 12:48:11 +0000
Mixlib::ShellOut::ShellCommandFailed: execute[create_admin] (firezone::create_admin line 23) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of bin/firezone rpc “FzHttp.Release.create_admin_user” ----
STDOUT:
STDERR: --rpc-eval : RPC failed with reason :nodedown
---- End output of bin/firezone rpc “FzHttp.Release.create_admin_user” ----
Ran bin/firezone rpc “FzHttp.Release.create_admin_user” returned 1
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in invalid!' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in error!’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in shell_out_compacted!' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in shell_out!’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider/execute.rb:52:in block (2 levels) in <class:Execute>' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/mixin/why_run.rb:51:in add_action’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider.rb:265:in converge_by' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider/execute.rb:50:in block in class:Execute’
(eval):2:in block in action_run' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider.rb:276:in instance_eval’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider.rb:276:in compile_and_converge_action' (eval):2:in action_run’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/provider.rb:217:in run_action' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource.rb:599:in block in run_action’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource.rb:626:in with_umask' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource.rb:598:in run_action’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:74:in run_action' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:108:in block in run_all_actions’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:108:in each' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:108:in run_all_actions’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:132:in block in converge' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/resource_list.rb:96:in block in execute_each_resource’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/stepable_iterator.rb:85:in step’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/stepable_iterator.rb:54:in each_with_index’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/resource_collection/resource_list.rb:94:in execute_each_resource' /opt/firezone/embedded/lib/ruby/2.7.0/forwardable.rb:235:in execute_each_resource’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/runner.rb:130:in converge' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/client.rb:687:in block in converge’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/client.rb:682:in catch' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/client.rb:682:in converge’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/client.rb:706:in converge_and_save' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/client.rb:286:in run’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/application.rb:305:in run_with_graceful_exit_option' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/application.rb:281:in block in run_chef_client’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/local_mode.rb:42:in with_server_connectivity' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/application.rb:264:in run_chef_client’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/application/base.rb:337:in run_application' /opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-16.17.51/lib/chef/application.rb:67:in run’
/opt/firezone/embedded/lib/ruby/gems/2.7.0/gems/chef-bin-16.17.51/bin/chef-client:25:in <top (required)>' /opt/firezone/embedded/bin/chef-client:23:in load’
So that command is failing because the phoenix service isn’t up. Could you post the output of firezone-ctl tail phoenix? That should uncover the issue.
Specs on that lightsail look fine.
root@ip-172-26-15-123:/var/opt/firezone# firezone-ctl tail phoenix
2022-07-20_14:20:32.02275 14:20:32.021 [notice] Application fz_http exited: shutdown
2022-07-20_14:20:33.58902 {“Kernel pid terminated”,application_controller,"{application_te
rminated,fz_http,shutdown}"}
2022-07-20_14:20:33.58948 Kernel pid terminated (application_controller) ({application_ter
minated,fz_http,shutdown})
2022-07-20_14:20:33.58949
2022-07-20_14:20:33.58949 Crash dump is being written to: erl_crash.dump…done
2022-07-20_14:20:36.36457 received TERM from runit, forcing quit
2022-07-20_14:20:39.23072 14:20:39.228 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0
at 127.0.0.1:13000 (http)
2022-07-20_14:20:39.23301 14:20:39.232 [info] Access FzHttpWeb.Endpoint at https://18.220.
156.117
2022-07-20_14:20:51.61058 14:20:51.606 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0
at 127.0.0.1:13000 (http)
2022-07-20_14:20:51.61272 14:20:51.612 [info] Access FzHttpWeb.Endpoint at https://18.220.
156.117
Hmmm. I think that’s clipping out the cause of the crash. Would you be able to post the full log (or at least the last 100 or 200 lines) from /var/log/firezone/phoenix/current?
Unfortunately there does not seem to be a ‘current’ directory in /var/log/firezone/phoenix/
It should be a plaintext file. Is it not there?
Oops, i must have missed it the first time round for some reason. Here is the full contents of the file.
root@ip-172-26-15-123:/var/log/firezone/phoenix# cat current
2022-07-20_14:20:31.82826 14:20:31.825 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:13000 (http)
2022-07-20_14:20:31.83304 14:20:31.832 [info] Access FzHttpWeb.Endpoint at https://18.220.156.117
2022-07-20_14:20:32.01189 14:20:31.986 [error] GenServer #PID<0.446.0> terminating
2022-07-20_14:20:32.01191 ** (Postgrex.Error) ERROR 42P01 (undefined_table) relation “users” does not exist
2022-07-20_14:20:32.01191
2022-07-20_14:20:32.01191 query: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“sign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FROM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_14:20:32.01192 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
2022-07-20_14:20:32.01192 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:847: Ecto.Adapters.SQL.execute/6
2022-07-20_14:20:32.01192 (ecto 3.8.4) lib/ecto/repo/queryable.ex:221: Ecto.Repo.Queryable.execute/4
2022-07-20_14:20:32.01192 (ecto 3.8.4) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
2022-07-20_14:20:32.01193 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:41: FzHttp.OIDC.RefreshManager.spawn_refresher/0
2022-07-20_14:20:32.01193 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:24: FzHttp.OIDC.RefreshManager.handle_continue/2
2022-07-20_14:20:32.01193 (stdlib 4.0.1) gen_server.erl:1120: :gen_server.try_dispatch/4
2022-07-20_14:20:32.01194 (stdlib 4.0.1) gen_server.erl:862: :gen_server.loop/7
2022-07-20_14:20:32.01194 Last message: {:continue, :schedule}
2022-07-20_14:20:32.01358 14:20:32.012 [error] GenServer #PID<0.452.0> terminating
2022-07-20_14:20:32.01359 ** (Postgrex.Error) ERROR 42P01 (undefined_table) relation “users” does not exist
2022-07-20_14:20:32.01359
2022-07-20_14:20:32.01359 query: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“sign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FROM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_14:20:32.01360 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
2022-07-20_14:20:32.01360 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:847: Ecto.Adapters.SQL.execute/6
2022-07-20_14:20:32.01360 (ecto 3.8.4) lib/ecto/repo/queryable.ex:221: Ecto.Repo.Queryable.execute/4
2022-07-20_14:20:32.01360 (ecto 3.8.4) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
2022-07-20_14:20:32.01361 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:41: FzHttp.OIDC.RefreshManager.spawn_refresher/0
2022-07-20_14:20:32.01361 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:24: FzHttp.OIDC.RefreshManager.handle_continue/2
2022-07-20_14:20:32.01361 (stdlib 4.0.1) gen_server.erl:1120: :gen_server.try_dispatch/4
2022-07-20_14:20:32.01365 (stdlib 4.0.1) gen_server.erl:862: :gen_server.loop/7
2022-07-20_14:20:32.01365 Last message: {:continue, :schedule}
2022-07-20_14:20:32.01492 14:20:32.014 [error] GenServer #PID<0.453.0> terminating
2022-07-20_14:20:32.01492 ** (Postgrex.Error) ERROR 42P01 (undefined_table) relation “users” does not exist
2022-07-20_14:20:32.01493
2022-07-20_14:20:32.01493 query: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“sign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FROM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_14:20:32.01493 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
2022-07-20_14:20:32.01494 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:847: Ecto.Adapters.SQL.execute/6
2022-07-20_14:20:32.01494 (ecto 3.8.4) lib/ecto/repo/queryable.ex:221: Ecto.Repo.Queryable.execute/4
2022-07-20_14:20:32.01494 (ecto 3.8.4) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
2022-07-20_14:20:32.01495 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:41: FzHttp.OIDC.RefreshManager.spawn_refresher/0
2022-07-20_14:20:32.01496 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:24: FzHttp.OIDC.RefreshManager.handle_continue/2
2022-07-20_14:20:32.01496 (stdlib 4.0.1) gen_server.erl:1120: :gen_server.try_dispatch/4
2022-07-20_14:20:32.01497 (stdlib 4.0.1) gen_server.erl:862: :gen_server.loop/7
2022-07-20_14:20:32.01497 Last message: {:continue, :schedule}
2022-07-20_14:20:32.01633 14:20:32.015 [error] GenServer #PID<0.454.0> terminating
2022-07-20_14:20:32.01634 ** (Postgrex.Error) ERROR 42P01 (undefined_table) relation “users” does not exist
2022-07-20_14:20:32.01634
2022-07-20_14:20:32.01634 query: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“sign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FROM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_14:20:32.01635 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
2022-07-20_14:20:32.01635 (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:847: Ecto.Adapters.SQL.execute/6
2022-07-20_14:20:32.01635 (ecto 3.8.4) lib/ecto/repo/queryable.ex:221: Ecto.Repo.Queryable.execute/4
2022-07-20_14:20:32.01635 (ecto 3.8.4) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
2022-07-20_14:20:32.01636 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:41: FzHttp.OIDC.RefreshManager.spawn_refresher/0
2022-07-20_14:20:32.01636 (fz_http 0.4.5) lib/fz_http/oidc/refresh_manager.ex:24: FzHttp.OIDC.RefreshManager.handle_continue/2
2022-07-20_14:20:32.01636 (stdlib 4.0.1) gen_server.erl:1120: :gen_server.try_dispatch/4
2022-07-20_14:20:32.01637 (stdlib 4.0.1) gen_server.erl:862: :gen_server.loop/7
2022-07-20_14:20:32.01638 Last message: {:continue, :schedule}
2022-07-20_14:20:32.02275 14:20:32.021 [notice] Application fz_http exited: shutdown
2022-07-20_14:20:33.58902 {“Kernel pid terminated”,application_controller,"{application_terminated,fz_http,shutdown}"}
2022-07-20_14:20:33.58948 Kernel pid terminated (application_controller) ({application_terminated,fz_http,shutdown})
2022-07-20_14:20:33.58949
2022-07-20_14:20:33.58949 Crash dump is being written to: erl_crash.dump…done
2022-07-20_14:20:36.36457 received TERM from runit, forcing quit
2022-07-20_14:20:39.23072 14:20:39.228 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:13000 (http)
2022-07-20_14:20:39.23301 14:20:39.232 [info] Access FzHttpWeb.Endpoint at https :// 18.220.156.117
2022-07-20_14:20:51.61058 14:20:51.606 [info] Running FzHttpWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:13000 (http)
2022-07-20_14:20:51.61272 14:20:51.612 [info] Access FzHttpWeb.Endpoint at https://18.220.156.117
root@ip-172-26-15-123:/var/log/firezone/phoenix#
That’s very strange and the first time I’m seeing this particular issue. Looks like the database migration failed and the necessary tables were never created.
If this is happening consistently when you try to install Firezone on this type of Lightsail, it could be the 512m memory. You’ll most likely need swap enabled to prevent getting OOM errors.
Is there anything relevant in /var/log/firezone/postgres/current?
Could you summarize any changes (esp DB changes) you’ve made to your configuration file?
I have made no configuration changes. This is a fresh install using the deployment instructions and I can confirm this happens consistently when I try to install using that type of LightSail
Here is the output for the postgres log file
root@ip-172-26-10-233:/var/log/firezone/postgresql# cat current
2022-07-20_15:49:55.68348 2022-07-20 15:49:55.683 GMT [2271] LOG: starting PostgreSQL 13.5 on x86_6 4-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2022-07-20_15:49:55.68361 2022-07-20 15:49:55.683 GMT [2271] LOG: listening on IPv4 address “127.0. 0.1”, port 15432
2022-07-20_15:49:55.68928 2022-07-20 15:49:55.689 GMT [2271] LOG: listening on Unix socket “/tmp/.s .PGSQL.15432”
2022-07-20_15:49:55.69857 2022-07-20 15:49:55.698 GMT [2272] LOG: database system was shut down at 2022-07-20 15:49:51 GMT
2022-07-20_15:49:55.70387 2022-07-20 15:49:55.703 GMT [2271] LOG: database system is ready to accep t connections
2022-07-20_15:50:16.55566 2022-07-20 15:50:16.555 GMT [2427] ERROR: relation “users” does not exist at character 231
2022-07-20_15:50:16.55639 2022-07-20 15:50:16.555 GMT [2427] STATEMENT: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“s ign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FR OM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_15:50:16.61770 2022-07-20 15:50:16.616 GMT [2428] ERROR: relation “users” does not exist at character 231
2022-07-20_15:50:16.61772 2022-07-20 15:50:16.616 GMT [2428] STATEMENT: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“s ign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FR OM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_15:50:16.62049 2022-07-20 15:50:16.620 GMT [2429] ERROR: relation “users” does not exist at character 231
2022-07-20_15:50:16.62050 2022-07-20 15:50:16.620 GMT [2429] STATEMENT: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“s ign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FR OM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_15:50:16.62489 2022-07-20 15:50:16.624 GMT [2430] ERROR: relation “users” does not exist at character 231
2022-07-20_15:50:16.62490 2022-07-20 15:50:16.624 GMT [2430] STATEMENT: SELECT u0.“id”, u0.“uuid”, u0.“role”, u0.“email”, u0.“last_signed_in_at”, u0.“last_signed_in_method”, u0.“password_hash”, u0.“s ign_in_token”, u0.“sign_in_token_created_at”, u0.“disabled_at”, u0.“inserted_at”, u0.“updated_at” FR OM “users” AS u0 WHERE (u0.“disabled_at” IS NULL)
2022-07-20_15:50:20.02077 received TERM from runit, sending INT instead to force quit connections
2022-07-20_15:50:20.02314 2022-07-20 15:50:20.023 GMT [2271] LOG: received fast shutdown request
2022-07-20_15:50:20.02608 2022-07-20 15:50:20.026 GMT [2271] LOG: aborting any active transactions
2022-07-20_15:50:20.02771 2022-07-20 15:50:20.027 GMT [2271] LOG: background worker “logical replic ation launcher” (PID 2278) exited with exit code 1
2022-07-20_15:50:20.02771 2022-07-20 15:50:20.027 GMT [2273] LOG: shutting down
2022-07-20_15:50:20.06415 2022-07-20 15:50:20.064 GMT [2271] LOG: database system is shut down
2022-07-20_15:50:20.10913 2022-07-20 15:50:20.109 GMT [2569] LOG: starting PostgreSQL 13.5 on x86_6 4-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2022-07-20_15:50:20.11025 2022-07-20 15:50:20.110 GMT [2569] LOG: listening on IPv4 address “127.0. 0.1”, port 15432
2022-07-20_15:50:20.11490 2022-07-20 15:50:20.114 GMT [2569] LOG: listening on Unix socket “/tmp/.s .PGSQL.15432”
2022-07-20_15:50:20.12299 2022-07-20 15:50:20.122 GMT [2570] LOG: database system was shut down at 2022-07-20 15:50:20 GMT
2022-07-20_15:50:20.12939 2022-07-20 15:50:20.129 GMT [2569] LOG: database system is ready to accep t connections
Did you install from the one-line install script? At some point that must have failed to run migrations, but that original error is not present in these logs.
What happens if you do a firezone-ctl reconfigure?
Yes I used the one-line install script.
bash <(curl -Ls https://github.com/firezone/firezone/raw/master/scripts/install.sh)
Running firezone-ctl reconfigure seems to run fine with out any error. I then further ran firezone-ctl create-or-reset-admin but I get the same error as before.
Hi @ryan – Does this instance have swap enabled? Any OOM errors that might be related to the reconfigure script running? Any process killed errors in your kernel log?
The error you’re seeing means the DB migrations didn’t finish. These happen during reconfigure.
Just a quick update – these are likely OOM errors crashing the BEAM and not being picked up by the reconfigure. For these small instances I’d make sure you have at least a little swap turned on.