Tower 3.8.3 installation fails on TASK pulp.pulp_installer.pulp_database_config

[Eddy]

I'm installing Ansible Tower on RHEL 8.4 with AutoHub on a separate VM also 8.4.

The install fails at the following - any ideas?
Doesn't seem to be firewall since I ran it with firewalld stopped.

TASK [pulp.pulp_installer.pulp_database_config : Run database migrations] ****************************
fatal: [autohub]: FAILED! => {"changed": true, "cmd": ["/usr/bin/pulpcore-manager", "migrate", "--no-input"], "delta": "0:00:02.314739", "end": "2021-07-26 13:46:51.958933", "msg": "non-zero return code", "rc": 1, "start": "2021-07-26 13:46:49.644194", "stderr": "Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect\n    self.connection = self.get_new_connection(conn_params)\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection\n    connection = Database.connect(**conn_params)\n  File \"/usr/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\npsycopg2.OperationalError: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/bin/pulpcore-manager\", line 11, in <module>\n    load_entry_point('pulpcore==3.7.5', 'console_scripts', 'pulpcore-manager')()\n  File \"/usr/lib/python3.6/site-packages/pulpcore/app/manage.py\", line 11, in manage\n    execute_from_command_line(sys.argv)\n  File \"/usr/lib/python3.6/site-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line\n    utility.execute()\n  File \"/usr/lib/python3.6/site-packages/django/core/management/__init__.py\", line 375, in execute\n    self.fetch_command(subcommand).run_from_argv(self.argv)\n  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 323, in run_from_argv\n    self.execute(*args, **cmd_options)\n  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 364, in execute\n    output = self.handle(*args, **options)\n  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 83, in wrapped\n    res = handle_func(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/django/core/management/commands/migrate.py\", line 87, in handle\n    executor = MigrationExecutor(connection, self.migration_progress_callback)\n  File \"/usr/lib/python3.6/site-packages/django/db/migrations/executor.py\", line 18, in __init__\n    self.loader = MigrationLoader(self.connection)\n  File \"/usr/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 49, in __init__\n    self.build_graph()\n  File \"/usr/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 212, in build_graph\n    self.applied_migrations = recorder.applied_migrations()\n  File \"/usr/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 73, in applied_migrations\n    if self.has_table():\n  File \"/usr/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 56, in has_table\n    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 256, in cursor\n    return self._cursor()\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 233, in _cursor\n    self.ensure_connection()\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__\n    raise dj_exc_value.with_traceback(traceback) from exc_value\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection\n    self.connect()\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect\n    self.connection = self.get_new_connection(conn_params)\n  File \"/usr/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection\n    connection = Database.connect(**conn_params)\n  File \"/usr/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect\n    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)\ndjango.db.utils.OperationalError: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection", "    self.connect()", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect", "    self.connection = self.get_new_connection(conn_params)", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection", "    connection = Database.connect(**conn_params)", "  File \"/usr/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect", "    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)", "psycopg2.OperationalError: could not connect to server: No such file or directory", "\tIs the server running locally and accepting", "\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?", "", "", "The above exception was the direct cause of the following exception:", "", "Traceback (most recent call last):", "  File \"/usr/bin/pulpcore-manager\", line 11, in <module>", "    load_entry_point('pulpcore==3.7.5', 'console_scripts', 'pulpcore-manager')()", "  File \"/usr/lib/python3.6/site-packages/pulpcore/app/manage.py\", line 11, in manage", "    execute_from_command_line(sys.argv)", "  File \"/usr/lib/python3.6/site-packages/django/core/management/__init__.py\", line 381, in execute_from_command_line", "    utility.execute()", "  File \"/usr/lib/python3.6/site-packages/django/core/management/__init__.py\", line 375, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 323, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 364, in execute", "    output = self.handle(*args, **options)", "  File \"/usr/lib/python3.6/site-packages/django/core/management/base.py\", line 83, in wrapped", "    res = handle_func(*args, **kwargs)", "  File \"/usr/lib/python3.6/site-packages/django/core/management/commands/migrate.py\", line 87, in handle", "    executor = MigrationExecutor(connection, self.migration_progress_callback)", "  File \"/usr/lib/python3.6/site-packages/django/db/migrations/executor.py\", line 18, in __init__", "    self.loader = MigrationLoader(self.connection)", "  File \"/usr/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 49, in __init__", "    self.build_graph()", "  File \"/usr/lib/python3.6/site-packages/django/db/migrations/loader.py\", line 212, in build_graph", "    self.applied_migrations = recorder.applied_migrations()", "  File \"/usr/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 73, in applied_migrations", "    if self.has_table():", "  File \"/usr/lib/python3.6/site-packages/django/db/migrations/recorder.py\", line 56, in has_table", "    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 256, in cursor", "    return self._cursor()", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 233, in _cursor", "    self.ensure_connection()", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection", "    self.connect()", "  File \"/usr/lib/python3.6/site-packages/django/db/utils.py\", line 89, in __exit__", "    raise dj_exc_value.with_traceback(traceback) from exc_value", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 217, in ensure_connection", "    self.connect()", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/base/base.py\", line 195, in connect", "    self.connection = self.get_new_connection(conn_params)", "  File \"/usr/lib/python3.6/site-packages/django/db/backends/postgresql/base.py\", line 178, in get_new_connection", "    connection = Database.connect(**conn_params)", "  File \"/usr/lib64/python3.6/site-packages/psycopg2/__init__.py\", line 127, in connect", "    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)", "django.db.utils.OperationalError: could not connect to server: No such file or directory", "\tIs the server running locally and accepting", "\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?"], "stdout": "", "stdout_lines": []}

[Fran_Garcia]

hi Eddy,

Unfortunately the error looks a bit non-specific... it seems that postgres didn't properly start in the node. Can you check whether it's started? 

Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/postgresql.service.d
└─override.conf
Active: active (running) since Sun 2021-07-25 14:13:10 EDT; 17h ago
Main PID: 894 (postmaster)
Tasks: 29 (limit: 204115)
Memory: 147.8M
CGroup: /system.slice/postgresql.service
├─ 894 /usr/bin/postmaster -D /var/lib/pgsql/data
├─ 917 postgres: logger
├─ 923 postgres: checkpointer
├─ 924 postgres: background writer
├─ 925 postgres: walwriter
├─ 926 postgres: autovacuum launcher
├─ 927 postgres: stats collector
├─ 928 postgres: logical replication launcher
├─ 1103 postgres: awx awx 192.168.130.245(53764) idle
├─ 1105 postgres: awx awx 192.168.130.245(53766) idle
├─ 1515 postgres: automationhub automationhub 192.168.130.240(49146) idle
└─ 1516 postgres: automationhub automationhub 192.168.130.240(49148) idle

If it's not running, I'd check any previous errors, and ensure the prerequisites are met (cpu, memory, latest version of selinux policy packages, etc). Also ensure no system hardening is preventing the installer to work successfully (umask, /tmp mount settings, others).

If this doesn't help I suggest opening a support case as the issue might be system-specific.

[Eddy]

The service is running.

The firewall is opened on the listening port (and tested with firewalld stopped as well).

The VM had Tower version 3.6.3 which cailed to upgrade so I am trying to install 3.8.3 on the same VM.

# systemctl status postgresql
● postgresql.service - PostgreSQL database server
Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/postgresql.service.d
└─override.conf
Active: active (running) since Mon 2021-07-26 13:02:57 IDT; 3h 48min ago
Main PID: 39174 (postmaster)
Tasks: 16 (limit: 24784)
Memory: 76.9M
CGroup: /system.slice/postgresql.service
├─39174 /usr/bin/postmaster -D /var/lib/pgsql/data
├─39175 postgres: logger process
├─39177 postgres: checkpointer process
├─39178 postgres: writer process
├─39179 postgres: wal writer process
├─39180 postgres: autovacuum launcher process
├─39181 postgres: stats collector process
├─39182 postgres: bgworker: logical replication launcher
├─55713 postgres: awx awx 127.0.0.1(48144) idle
├─55720 postgres: awx awx 127.0.0.1(48148) idle
├─55721 postgres: awx awx 127.0.0.1(48150) idle
├─56068 postgres: awx awx 127.0.0.1(48168) idle
├─61169 postgres: awx awx 127.0.0.1(53602) idle
├─61170 postgres: awx awx 127.0.0.1(53604) idle
├─61220 postgres: awx awx 127.0.0.1(53612) idle
└─61225 postgres: awx awx 127.0.0.1(53616) idle

Jul 26 13:02:57 ansible-tower.demo.com systemd[1]: Starting PostgreSQL database server...
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.158 UTC >LOG: listening on IPv>
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.158 UTC >LOG: listening on IPv>
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.160 UTC >LOG: listening on Uni>
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.164 UTC >LOG: listening on Uni>
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.187 UTC >LOG: redirecting log >
Jul 26 13:02:57 ansible-tower.demo.com postmaster[39174]: < 2021-07-26 10:02:57.187 UTC >HINT: Future log outp>
Jul 26 13:02:57 ansible-tower.demo.com systemd[1]: Started PostgreSQL database server.

# netstat -nap|grep postgres
tcp 0 0 127.0.0.1:5432 127.0.0.1:48168 ESTABLISHED 56068/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:53684 ESTABLISHED 61292/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:48144 ESTABLISHED 55713/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:53696 ESTABLISHED 61298/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:48150 ESTABLISHED 55721/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:53692 ESTABLISHED 61296/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:48148 ESTABLISHED 55720/postgres: awx
tcp 0 0 127.0.0.1:5432 127.0.0.1:53688 ESTABLISHED 61294/postgres: awx
unix 2 [ ACC ] STREAM LISTENING 229429 39174/postmaster /var/run/postgresql/.s.PGSQL.5432
unix 3 [ ] STREAM CONNECTED 229421 39175/postgres: log

# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: cockpit dhcpv6-client ssh
ports: 80/tcp 443/tcp 5432/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

[Deanna]

Hi @Eddy - did the installation of 3.8.3 on the VM solve the error? 

[Eddy]

The error comes form the installation of version 3.8.3

Version 3.6.3 was working fine ...

[theAeon]

Seconding this-only difference is that I'm attempting a clean install. Same error. Worth noting that my install is reporting the fail on the autohub host rather than the tower host (you know, the one with the database)-is it possible that its trying to connect to postgres on the wrong machine?

EDIT: 3.8.4

EDIT 2: manually defining the database as the IP of my tower node fixed it.

[SteveBisnett · Accepted solution]

Did you find a fix to this issue? I am experiencing the same issue even after disabling the firewalld on BOTH system. 

Just saw your EDIT2.. Sorry!

[Noel]

I had the same issue when installing Automation Controller - Bundle version 2.1.2-1

OS RHEL 8.5

The issue is that pulp is unable to connect to PostgreSQL using the username and password specified.

As a workaround I followed the below steps:

# export PULP_SETTINGS='/etc/pulp/settings.py'

check firewall and configure properly

then:

# dnf module install postgresql

# postgresql-setup --initdb

# systemctl start postgresql.service

add the below line inside the configuration file /var/lib/pgsql/data/postgresql.conf

listen_addresses = '*'

update the file /var/lib/pgsql/data/pg_hba.conf with the below configuration

host all all 0.0.0.0/0 md5
host all::/0 md5

Then:

# systemctl restart postgresql.service

Now create the database, user, user-password as specified in install-inventory file

# su - postgres

The below command will ask for role(username) and password

# createuser --interactive --pwprompt

# psql

execute the below commands inside the interactive shell of psql

# CREATE DATABASE automationhub;

my user and database is the same "automationhub", so change your values accordingly

# grant ALL privileges on database automationhub to automationhub;

exit from psql

# \q

Now run the below command on the automation hub VM to verify if it worked or not.

# pulpcore-manager migrate --no-input

If this worked fine then re-execute setup.sh

[Fran_Garcia]

It really seems your inventory file is wrong, you should not need to manually configure a database.

Note that when deploying an all in one controller+hub, the HUB database is run in the controller machine. If you have a external database server, both controller and hub query the PostgreSQL DB running in the db system.

E.g.:

# cat inventory | grep pg_host
pg_host='aap_controller.example.org'
automationhub_pg_host='aap_controller.example.org'

Neither of those two instances should be blank, or pointing to 'localhost'.

[Noel]

Thanks @Fran_Garcia