Environment

• YugabyteDB Anywhere - 2.20.x

Issue

Whilst attempting to update the flags through the YBA GUI the following is showing the following: 

Failed to execute task {"platformVersion":"2.20.6.0-b66","sleepAfterMasterRestartMillis":180000,"sleepAfterTServerRestartMillis":180000,"nodeExporterUser":"prometheus","deviceInfo":{"volumeSize":300,"numVolumes":1,"storageClass":"standard","mountPoints":"/yb01"},"universeUUID":"<UNIVERSE_UUID>","enableYbc":true,"ybcSoftwareVersion":"2.1.0.0-b9","installYbc":true,"ybcInstalled":true,"encryptionAtRestConfig":{"encryptionAtRestEnabled":false,"opType":"UNDEFINED","type":"DATA_KEY"},"communicationPo..., hit error:

Error occurred. Code: 3. Output: ybops.common.exceptions.YBOpsRecoverableError: Custom exit code exception: Playbook run of configure-cluster-server.yml against REDACTED, with args ['/opt/yugabyte/[PATH_TO]/ansible-playbook.sh', '/opt/yugabyte/[PATH_TO]/configure-cluster-server.yml', '--private-key', '/opt/yugabyte/[PATH_TO]/qa-key.pem', '--vault-password-file', '/opt/yugabyte/[PATH_TO]/qa-key.vault_password', '--tags', 'override_gflags', '--user', 'yugabyte', '-i', 'REDACTED,', '-c', 'ssh', '--extra-vars', '{"user_name": "yugabyte", "instance_search_pattern": "all", "cloud_type": "onprem", "cloud_region": "[REGION]", "cloud_zone": "[ZONE]", "custom_ssh_port": "22", "instance_name": "yb-stage-instance-n3", "master_addresses_for_tserver": "[REDACTED]:7100,[REDACTED]:7100,[REDACTED]:7100", "master_http_port": "7000", "master_rpc_port": "7100", "tserver_http_port": "9000", "tserver_rpc_port": "9100", "cql_proxy_rpc_port": "9042", "redis_proxy_rpc_port": "6379", "cert_valid_duration": 365, "org_name": "example.com", "certs_client_dir": null, "certs_node_dir": "/apps/yugabyte/yugabyte-tls-config", "encryption_key_dir": "REDACTED", "master_addresses_for_master": "[REDACTED]", "acceptable_clock_skew_max_tries": "300", "private_ip": "REDACTED", "public_ip": "REDACTED", "placement_cloud": "onprem", "placement_region": "REDACTED", "placement_zone": "REDACTED", "instance_type": "vmh_16x128", "host": "REDACTED", "port": "22", "user": "yugabyte", "ssh_user": "yugabyte", "yb_server_ssh_user": "yugabyte", "ssh_version": "ssh", "yb_home_dir": "/apps/yugabyte", "yb_ansible_host": "REDACTED", "ansible_port": "22", "vars_file": "/opt/yugabyte/data/yb-platform/keys/REDACTED/qa-key.vault"}'] failed with return code 4 and error '[0;32m[2024-10-09_17_33_57 common.sh:650 activate_pex][0m Using pex virtualenv python executable now.
[WARNING]: Found variable using reserved name: port
[WARNING]: Unhandled error in Python interpreter discovery for host
REDACTED: Failed to connect to the host via ssh:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  @    WARNING:
REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  IT IS POSSIBLE
THAT SOMEONE IS DOING SOMETHING NASTY!  Someone could be eavesdropping on you
right now (man-in-the-middle attack)!  It is also possible that a host key has
just been changed.  The fingerprint for the ED25519 key sent by the remote host
is SHA512:REDACTED.  Please contact your system administrator.  Host key
verification failed.

To check the error, attempt to ssh to the node - the same error will be shown.

Resolution

Overview

Firstly, check that the remote host is actually what you think it is and that there has been some patching or some change that could affect the node (e.g. the host has a different ip address or a different name).

Remove the relevant entry from the known_hosts, add the new entry and check using ssh that the issue is no longer there.

Steps

1. Remove the entry for the host that is shown in the error message using ssh-keygen
 

ssh-keygen -R <host>[:port]

e.g.

ssh-keygen -R 123.45.67.8

or, for a specific port (In this example, port 24)
 

ssh-keygen -R 123.45.67.8:24

Note: The use of a port number is only required if the default ssh port (22) has been reassigned and you connect using ssh via a different port

-  OR -

Remove the entry from the known_hosts file and reconnect

known_hosts is typically stored in the ~/.ssh directory, where ~ is the current user.

cd <PATH TO known_hosts parent directory>
vi known_hosts
delete the line(s) that contain the host that is complaining
save the file

Once you have completed either of those steps, test by connecting via ssh - if you no longer see the error message, the issue is fixed.

When you first connect, you will be prompted to add the new key back into the known_hosts file.

NOTE: Ensure that there is not a man in the middle attack attempt and that the cause is something as innocuous as a changed hostname or ip address or a patched operating system.