vpxd-svcs startet nicht mehr

von René Gessinger · Veröffentlicht 7. Juni 2021 · Aktualisiert 8. Juni 2021

Ein vCenter eines Kunden hatte heute morgen nicht mehr funktioniert. Der WebClient zeigte lediglich die Meldung no healthy upstream und auf die API konnte man nicht verbinden.

Zuerst einmal schaute ich mir an welche Dienste gestartet waren und welche nicht:

# service-control --status

Running:

applmgmt lookupsvc lwsmd observability vmafdd vmcad vmdird vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-envoy vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-trustmanagement vmware-vdtc vmware-vmon vmware-vpostgres vsphere-ui vtsdb

Stopped:

observability-vapi pschealth vlcm vmcam vmware-certificateauthority vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vstats wcp

# service-control --status

Running:

applmgmt lookupsvc lwsmd observability vmafdd vmcad vmdird vmonapi vmware-analytics vmware-cis-license vmware-eam vmware-envoy vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-trustmanagement vmware-vdtc vmware-vmon vmware-vpostgres vsphere-ui vtsdb

Stopped:

observability-vapi pschealth vlcm vmcam vmware-certificateauthority vmware-certificatemanagement vmware-content-library vmware-hvc vmware-imagebuilder vmware-infraprofile vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vapi-endpoint vmware-vcha vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vstats wcp

Dort sieht das z.B. die Datenbank und andere Core-Services gestartet waren, aber z.B. vpxd (der vCenter Server Daemon) und vpxd-svcs nicht. Da vpxd von vpxd-svcs abhängig ist, sollte man überprüfen warum dieser nicht startet. Hilfreich ist dabei das Logfile dieses Dienstens. Dies ist zu finden unter /storage/log/vmware/vpxd-svcs/vpxd-svcs.log.
Dort befand sich der folgende Eintrag:

2021-06-07T17:14:34.232+02:00 [Thread-13  ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl  opId=] SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
    at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
    at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116)
    at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259)
    at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)
    at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161)
    at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:965)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:894)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:501)
    at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:52)
    at com.vmware.cis.server.ssoauthentication.impl.AbstractTokenProvider.getSamlToken(AbstractTokenProvider.java:42)
    at com.vmware.cis.server.util.VpxdClient.loginBySamlToken(VpxdClient.java:176)
    at com.vmware.cis.server.util.VpxdClient.login(VpxdClient.java:73)
    at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:155)
    at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:145)
    at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1691)
    at com.vmware.cis.server.util.impl.InitPoolTask.run(InitPoolTask.java:44)
    at java.lang.Thread.run(Thread.java:748)
2021-06-07T17:14:34.233+02:00 [Thread-13  INFO  com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor  opId=] Provided credentials are not valid.
2021-06-07T17:14:34.233+02:00 [Thread-13  WARN  com.vmware.cis.server.util.impl.InitPoolTask  opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 11

2021-06-07T17:14:34.232+02:00 [Thread-13 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] SOAP fault

com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.

at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)

at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116)

at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259)

at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)

at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161)

at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114)

at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:965)

at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:894)

at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:501)

at com.vmware.cis.server.ssoauthentication.impl.SolutionTokenProvider.acquireSamlToken(SolutionTokenProvider.java:52)

at com.vmware.cis.server.ssoauthentication.impl.AbstractTokenProvider.getSamlToken(AbstractTokenProvider.java:42)

at com.vmware.cis.server.util.VpxdClient.loginBySamlToken(VpxdClient.java:176)

at com.vmware.cis.server.util.VpxdClient.login(VpxdClient.java:73)

at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:155)

at com.vmware.cis.server.util.ConnectionManager$1.makeObject(ConnectionManager.java:145)

at org.apache.commons.pool.impl.GenericObjectPool.addObject(GenericObjectPool.java:1691)

at com.vmware.cis.server.util.impl.InitPoolTask.run(InitPoolTask.java:44)

at java.lang.Thread.run(Thread.java:748)

2021-06-07T17:14:34.233+02:00 [Thread-13 INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor opId=] Provided credentials are not valid.

2021-06-07T17:14:34.233+02:00 [Thread-13 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.AuthenticationException at attempt 11

Dies hat in den meisten Fällen einen von zwei möglichen Gründen:

die Trust Anchor sind fehlerhaft konfiguriert
es gibt abgelaufene Zertifikate in der Konfiguration der diversen Dienste

Den ersten Grund kann man über den lsdoctor (https://kb.vmware.com/s/article/80469) prüfen. Doch in diesem Fall brachte dies keine Probleme zum Vorschein (“no problems found”).
Um den zweiten Grund zu überprüfen, gibt es ein Shellcommand. Dieses listet alle Zertifikate auf und zeigt das Ablaufdatum an:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list);
    do echo STORE $i;
    sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After";
done

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list);

do echo STORE $i;

sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After";

done

Hier zeigte sich die folgende Ausgabe:

STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Apr  3 10:44:59 2022 GMT
STORE TRUSTED_ROOTS
Alias : 4bafe18534b46e9bd89b69297552480d60e5c8d1
            Not After : Jun 29 09:07:28 2028 GMT
Alias : b16aba94efa9dbeb7e89b357dd46f98401186b39
            Not After : Jun  1 07:05:39 2019 GMT
Alias : 90fb83b6f4e2db3c627bb2a013611e2bd453d917
            Not After : Jun  1 07:05:39 2019 GMT
Alias : c6a3954d301afae121e05d201f742bdb91363577
            Not After : Jun  1 06:01:42 2023 GMT
Alias : 4f97db26fe9bcf8fa570237089fc7ed750eb94c2
            Not After : Apr 11 15:52:46 2020 GMT
Alias : 826fb08ea790b0f6ff14194d4f29b47a15aa3aa8
            Not After : Apr 11 15:52:46 2020 GMT
Alias : 11b79606b01cc8989cf1aec8f72970bc4aef4bad
            Not After : Jun  1 12:46:12 2029 GMT
Alias : 58374b6569feb58c92807ac183cacd96fd071064
            Not After : Mar 19 12:08:34 2030 GMT
Alias : d745455353aa06aeb5460cbabd9233eeb21eee0a
            Not After : Mar 19 11:15:38 2040 GMT
STORE TRUSTED_ROOT_CRLS
Alias : 34689d1addeb1b79fd5f51798b9df3c5673315f3
Alias : 05858cac891c9305189b4c8438bd38baaa3c9d67
Alias : a683a2db1f38c48de4448659a39dc8ea8cdf11ff
Alias : 50ac04f1044f565ce33df971338d2e1732294a74
STORE machine
Alias : machine
            Not After : Jun  6 12:37:31 2021 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
            Not After : Jun  6 12:37:32 2021 GMT
STORE vpxd
Alias : vpxd
            Not After : Jun  6 12:37:32 2021 GMT
STORE vpxd-extension
Alias : vpxd-extension
            Not After : Jun  6 12:37:32 2021 GMT
STORE SMS
Alias : sms_self_signed
            Not After : Jul  5 09:12:20 2028 GMT
STORE BACKUP_STORE
Alias : bkp___MACHINE_CERT
            Not After : Apr 11 15:52:46 2020 GMT
Alias : bkp_machine
            Not After : Jun  6 12:37:31 2021 GMT
Alias : bkp_vsphere-webclient
            Not After : Jun  6 12:37:32 2021 GMT
Alias : bkp_vpxd
            Not After : Jun  6 12:37:32 2021 GMT
Alias : bkp_vpxd-extension
            Not After : Jun  6 12:37:32 2021 GMT
STORE APPLMGMT_PASSWORD
STORE data-encipherment
Alias : data-encipherment
            Not After : Jun  1 12:46:12 2029 GMT
STORE hvc
Alias : hvc
            Not After : Jun  1 12:46:12 2029 GMT
STORE wcp
Alias : wcp
            Not After : Sep  8 11:16:27 2022 GMT

STORE MACHINE_SSL_CERT

Alias : __MACHINE_CERT

Not After : Apr 3 10:44:59 2022 GMT

STORE TRUSTED_ROOTS

Alias : 4bafe18534b46e9bd89b69297552480d60e5c8d1

Not After : Jun 29 09:07:28 2028 GMT

Alias : b16aba94efa9dbeb7e89b357dd46f98401186b39

Not After : Jun 1 07:05:39 2019 GMT

Alias : 90fb83b6f4e2db3c627bb2a013611e2bd453d917

Not After : Jun 1 07:05:39 2019 GMT

Alias : c6a3954d301afae121e05d201f742bdb91363577

Not After : Jun 1 06:01:42 2023 GMT

Alias : 4f97db26fe9bcf8fa570237089fc7ed750eb94c2

Not After : Apr 11 15:52:46 2020 GMT

Alias : 826fb08ea790b0f6ff14194d4f29b47a15aa3aa8

Not After : Apr 11 15:52:46 2020 GMT

Alias : 11b79606b01cc8989cf1aec8f72970bc4aef4bad

Not After : Jun 1 12:46:12 2029 GMT

Alias : 58374b6569feb58c92807ac183cacd96fd071064

Not After : Mar 19 12:08:34 2030 GMT

Alias : d745455353aa06aeb5460cbabd9233eeb21eee0a

Not After : Mar 19 11:15:38 2040 GMT

STORE TRUSTED_ROOT_CRLS

Alias : 34689d1addeb1b79fd5f51798b9df3c5673315f3

Alias : 05858cac891c9305189b4c8438bd38baaa3c9d67

Alias : a683a2db1f38c48de4448659a39dc8ea8cdf11ff

Alias : 50ac04f1044f565ce33df971338d2e1732294a74

STORE machine

Alias : machine

Not After : Jun 6 12:37:31 2021 GMT

STORE vsphere-webclient

Alias : vsphere-webclient

Not After : Jun 6 12:37:32 2021 GMT

STORE vpxd

Alias : vpxd

Not After : Jun 6 12:37:32 2021 GMT

STORE vpxd-extension

Alias : vpxd-extension

Not After : Jun 6 12:37:32 2021 GMT

STORE SMS

Alias : sms_self_signed

Not After : Jul 5 09:12:20 2028 GMT

STORE BACKUP_STORE

Alias : bkp___MACHINE_CERT

Not After : Apr 11 15:52:46 2020 GMT

Alias : bkp_machine

Not After : Jun 6 12:37:31 2021 GMT

Alias : bkp_vsphere-webclient

Not After : Jun 6 12:37:32 2021 GMT

Alias : bkp_vpxd

Not After : Jun 6 12:37:32 2021 GMT

Alias : bkp_vpxd-extension

Not After : Jun 6 12:37:32 2021 GMT

STORE APPLMGMT_PASSWORD

STORE data-encipherment

Alias : data-encipherment

Not After : Jun 1 12:46:12 2029 GMT

STORE hvc

Alias : hvc

Not After : Jun 1 12:46:12 2029 GMT

STORE wcp

Alias : wcp

Not After : Sep 8 11:16:27 2022 GMT

Hier zeigten sich unter anderem im machine-Store Zertifikate welche abgelaufen waren.

Zur Fehlerbehebung kann man nun mittels dem Certificate Manager (https://kb.vmware.com/s/article/2097936) die Zertifikate zurücksetzen (Option 8).
Danach startete das vCenter wieder korrekt. Nun kann man wieder neue CA-Zertifikate implementieren.
Wie jede Änderung an den Zertifikaten, hat dies Einfluss an die Integration von Drittanbieterprogrammen. Dort muss der Trust evtl. neu hergestellt werden.

vpxd-svcs startet nicht mehr

Schreibe einen Kommentar Antworten abbrechen

Kategorien