Red Hat Satellite : A backend service [ Candlepin ] is unreachable

[DavidPoulin79]

Hi,

we are a new to Red Hat and just installed our first Red Hat Satellite server (Satellite 6.16 on RHEL 9.4). While try to importe the manifest to Satellite, we saw the message "A backend service [ Candlepin ] is unreachable" (in this url : /subscriptions) and we seem to not be able to upload manifest.

By trying to found some documentation to resolve it, we take a look a the "/about" and see the backend system and see that is indicate the below (you can see it in attachment i uploaded too)

Backend System Status
Component Status Message
candlepin FAIL Failed to open TCP connection to localhost:23443 (Connection refused - connect(2) for "localhost" port 23443)
candlepin_auth FAIL A backend service [ Candlepin ] is unreachable
foreman_tasks OK
katello_events OK 0 Processed, 0 Failed
candlepin_events FAIL Not running
pulp3 OK
pulp3_content OK

As a beginner, not sure where to search and i wonder if there something related to SeLinux or CIS profile i select during installation....i set SeLinux in permissive mode but didn't help..

When i tried some red hat article stuff to try to resolve it by myself (like this one: https://access.redhat.com/solutions/6977751?band=se&seSessionId=e0313874-8b14-4733-8462-c837fbff547a...) i can see some error i didn't saw on the first installation like these

sudo satellite-installer
2024-11-16 14:28:10 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-11-16 14:28:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-11-16 14:28:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Package versions are locked. Continuing with unlock.
2024-11-16 14:28:23 [NOTICE] [configure] Starting system configuration.
2024-11-16 14:28:32 [ERROR ] [configure] Failed to read keystore '/etc/candlepin/certs/keystore'
2024-11-16 14:28:32 [NOTICE] [configure] 250 configuration steps out of 1526 steps complete.
2024-11-16 14:28:34 [NOTICE] [configure] 500 configuration steps out of 1528 steps complete.
2024-11-16 14:28:35 [ERROR ] [configure] Failed to add certificate to keystore: Execution of '/bin/keytool -importkeystore -noprompt -srckeystore /tmp/temp_keystore20241116-262484-x3lygc -srcstorepass:file /etc/candlepin/certs/keystore_password-file -destkeystore /etc/candlepin/certs/keystore -deststorepass:file /etc/candlepin/certs/keystore_password-file -srcalias tomcat -destalias tomcat -J-Dcom.redhat.fips=false' returned 1: Importing keystore /tmp/temp_keystore20241116-262484-x3lygc to /etc/candlepin/certs/keystore...
2024-11-16 14:28:35 [ERROR ] [configure] keytool error: java.lang.Exception: Keystore file exists, but is empty: /etc/candlepin/certs/keystore
2024-11-16 14:28:36 [NOTICE] [configure] 750 configuration steps out of 1530 steps complete.
2024-11-16 14:28:37 [NOTICE] [configure] 1000 configuration steps out of 1536 steps complete.
2024-11-16 14:28:38 [NOTICE] [configure] 1250 configuration steps out of 1536 steps complete.
2024-11-16 14:32:15 [NOTICE] [configure] 1500 configuration steps out of 1536 steps complete.
2024-11-16 14:32:19 [NOTICE] [configure] System configuration has finished.
Success!
* Satellite is running at https://XXXXXXXXXX.com

* To install an additional Capsule on separate machine continue by running:

capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"
* Capsule is running at https://XXXXXXXXXX:9090

The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.

I also did the Hammer ping and get this :

hammer ping
database:
Status: ok
Server Response: Duration: 0ms
cache:
servers:
1) Status: ok
Server Response: Duration: 0ms
candlepin:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Connection refused - connect(2) for "localhost" port 23443)
candlepin_auth:
Status: FAIL
Server Response: Message: A backend service [ Candlepin ] is unreachable
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 1ms
pulp3:
Status: ok
Server Response: Duration: 92ms
pulp3_content:
Status: ok
Server Response: Duration: 84ms
foreman_tasks:
Status: ok
Server Response: Duration: 4ms

And from the developper panel in chrome and can api cal get a 500 error

Request URL:
https://XXXXXXXXXX/katello/api/v2/organizations/1/subscriptions/manifest_history
Request Method:
GET
Status Code:
500 Internal Server Error
Remote Address:
xx.xx.xx.x:443
Referrer Policy:
strict-origin-when-cross-origin

So I would really appreciate some help to dig further and learn on that.

Thank in advance for your help on this.