Today I was upgrading a VCSA 6.5 instance to 6.7 as I’ve done a dozen times. It’s a really simple setup – embedded PSC, etc. However, I just couldn’t get it to work!

I was constantly presented with “Error in getting source info” from the UI and the upgrade logs contained:

2018-07-18T12:47:08.855Z - info: VM with hostName [redacted-vc-fqdn]: { _type: 'VirtualMachine', _moid: 'vm-961', config: { template: false }, guest: { hostName: '[redacted-vc-fqdn]', ipAddress: '10.20.10.38', net: [ [Object] ] }, name: 'VMware vCenter Server Appliance', summary: { storage: { committed: 257543841825, uncommitted: 246668061721, unshared: 246689054720, timestamp: 2018-07-18T12:19:52.857Z } } } 2018-07-18T12:47:08.856Z - info: VM Identifier for Source VC: vm-961 2018-07-18T12:47:09.268Z - debug: initiateFileTransferFromGuest error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.268Z - debug: Failed to get fileTransferInfo:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.269Z - debug: Failed to get url of file in guest vm:ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.269Z - error: Error in getting fileData for nodeType. Error: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.270Z - error: Failed to read the nodetype, Error: Failed to authenticate with the guest operating system using the supplied credentials. 2018-07-18T12:47:09.272Z - info: Checking if password expired 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 2018 - 07 - 18T12 : 47 : 08.855Z - info : VM with hostName [ redacted - vc - fqdn ] : { _type : 'VirtualMachine' , _moid : 'vm-961' , config : { template : false } , guest : { hostName : '[redacted-vc-fqdn]' , ipAddress : '10.20.10.38' , net : [ [ Object ] ] } , name : 'VMware vCenter Server Appliance' , summary : { storage : { committed : 257543841825 , uncommitted : 246668061721 , unshared : 246689054720 , timestamp : 2018 - 07 - 18T12 : 19 : 52.857Z } } } 2018 - 07 - 18T12 : 47 : 08.856Z - info : VM Identifier for Source VC : vm - 961 2018 - 07 - 18T12 : 47 : 09.268Z - debug : initiateFileTransferFromGuest error : ServerFaultCode : Failed to authenticate with the guest operating system using the supplied credentials . 2018 - 07 - 18T12 : 47 : 09.268Z - debug : Failed to get fileTransferInfo : ServerFaultCode : Failed to authenticate with the guest operating system using the supplied credentials . 2018 - 07 - 18T12 : 47 : 09.269Z - debug : Failed to get url of file in guest vm : ServerFaultCode : Failed to authenticate with the guest operating system using the supplied credentials . 2018 - 07 - 18T12 : 47 : 09.269Z - error : Error in getting fileData for nodeType . Error : ServerFaultCode : Failed to authenticate with the guest operating system using the supplied credentials . 2018 - 07 - 18T12 : 47 : 09.270Z - error : Failed to read the nodetype , Error : Failed to authenticate with the guest operating system using the supplied credentials . 2018 - 07 - 18T12 : 47 : 09.272Z - info : Checking if password expired

I know you’re thinking “well, yeah, it says ‘Checking if password expired’ …” but alas, it says it is not:

2018-07-18T12:47:12.549Z - info: STDOUT: * List APIs: "help api list" * List Plugins: "help pi list" * Launch BASH: "shell" 2018-07-18T12:47:12.551Z - info: STDOUT: Command> 2018-07-18T12:47:12.554Z - info: STDOUT: da 2018-07-18T12:47:12.555Z - info: STDOUT: t 2018-07-18T12:47:12.556Z - info: STDOUT: e 2018-07-18T12:47:12.567Z - info: STDOUT: Unknown command: `date' 2018-07-18T12:47:12.568Z - info: STDOUT: Command> 2018-07-18T12:47:12.568Z - info: STDOUT: ex 2018-07-18T12:47:12.569Z - info: STDOUT: i 2018-07-18T12:47:12.570Z - info: STDOUT: t 2018-07-18T12:47:12.652Z - info: Stream :: close 2018-07-18T12:47:12.652Z - info: Password not expired 2018-07-18T12:47:12.657Z - error: sourcePrecheck: error in getting source Info: ServerFaultCode: Failed to authenticate with the guest operating system using the supplied credentials. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 2018 - 07 - 18T12 : 47 : 12.549Z - info : STDOUT : * List APIs : "help api list" * List Plugins : "help pi list" * Launch BASH : "shell" 2018 - 07 - 18T12 : 47 : 12.551Z - info : STDOUT : Command > 2018 - 07 - 18T12 : 47 : 12.554Z - info : STDOUT : da 2018 - 07 - 18T12 : 47 : 12.555Z - info : STDOUT : t 2018 - 07 - 18T12 : 47 : 12.556Z - info : STDOUT : e 2018 - 07 - 18T12 : 47 : 12.567Z - info : STDOUT : Unknown command : ` date ' 2018 - 07 - 18T12 : 47 : 12.568Z - info : STDOUT : Command > 2018 - 07 - 18T12 : 47 : 12.568Z - info : STDOUT : ex 2018 - 07 - 18T12 : 47 : 12.569Z - info : STDOUT : i 2018 - 07 - 18T12 : 47 : 12.570Z - info : STDOUT : t 2018 - 07 - 18T12 : 47 : 12.652Z - info : Stream :: close 2018 - 07 - 18T12 : 47 : 12.652Z - info : Password not expired 2018 - 07 - 18T12 : 47 : 12.657Z - error : sourcePrecheck : error in getting source Info : ServerFaultCode : Failed to authenticate with the guest operating system using the supplied credentials .

There’s clearly an issue in either an API call or some mechanism checking the password expiration status since we’re seeing malformed STDOUT returns and the password is marked as not expired above.

Strange – the password is not expired… yet when I was in VAMI under Administration the password expiration showed that it had expired in 1970 which is usually an indicator that it has in fact expired. Oddly enough, though, I could SSH to and access the console of the VCSA with the known expired “not expired” password.

I ended up resetting the root user password using root@vc-fqdn [ ~ ]# passwd New password: where I reset the password to a temporary password, rebooted, and then reset the password back to the original password, rebooted, and then attempted the upgrade. At this point everything worked as expected.

Truly weird. If it were expired I’d have not been able to log in and would have had to use this VMware KB in order to reset the root credentials (I’ve had to do this before on other VCSA’s).

Debating on turning the original VCSA back on and checking the auth logs but… then again, I’d rather just move forward 🙂

Edit: I decided that pressing on would not be beneficial to the community. So, I resurrected the original VCSA and did some searching:

Turns out the root credential was in fact expired. Not sure why I was still able to login and SSH but that’s life.

Protip: You need to keep an eye on this (unlike I have!) because if the root account expired then the script (/etc/cron.daily/logrotate) that runs to prune logs will not successfully execute and you will fill up /dev/sda3 – learned this one the hard way many moons ago.

Hope you guys find this useful!

Share this: Twitter

Facebook

