SGX sealing is vulnerable to rollback attacks as the enclave is not able to tell if the sealed data is the latest or a old copy. To mitigate this attack, monotonic counter (MC) has been introduced in Intel SGX SDK 1.8. This post looks into some implementation details inside Intel SGX SDK.

1. sgx_create_monotonic_counter[1]

Look at the description of this API, two things show up – first, the counter creation involves non-volatile memory (NVM) writing; second, limited number of MCs can be created per enclave due to the quota. Apparently, unlike the older version of Intel SGX SDK, where MC was only available in the simulation mode, the MC here is based on real NVM, which also seems to have limited capacity. Let’s go deeper.

2. PSE (Platform Service/Security Enclave?)

The MC creation essentially calls sgx_create_monotonic_counter_ex[2] inside SDK. This function builds two pse_message_t messages for the request and reply, which are both passed to crypt_invoke. The crypt_invoke function establishes a PSE session (secure and authenticated channel) with aesmd, encrypts the request using the shared secret, and calls uae_invoke_service with both the request and reply passed, which eventually calls invoke_service_ocall[3] in the PSE provided by PSW other than the SDK.

3. OAL (OS Abstraction Layer?)

The PSE request and reply are then delivered to oal_invoke_service[4], which essentially transforms the PSE messages into AEInvokeServiceRequest and AEInvokeServiceResponse, and sends the AE messages to aesmd via UNIX socket ‘/var/run/aesmd/aesm.socket’. This MC creation request will then be handled by PSE_MC_SERVICE as a PSE_MC_CREATE command[1]

4. PSE_OP (PSE OPeration?)

The service handler table defined in pse_op.cpp[5] lists all the MC related command PSE_MC_SERVICE supports:

static const struct service_handler_t { uint16_t service_id; uint16_t service_cmd; uint16_t req_size; uint16_t resp_size; srv_pfn_t srv_pfn; } service_handler[] = { {PSE_MC_SERVICE, PSE_MC_CREATE, sizeof(pse_mc_create_req_t), sizeof(pse_mc_create_resp_t), pse_mc_create}, {PSE_MC_SERVICE, PSE_MC_READ, sizeof(pse_mc_read_req_t), sizeof(pse_mc_read_resp_t), pse_mc_read }, {PSE_MC_SERVICE, PSE_MC_INC, sizeof(pse_mc_inc_req_t), sizeof(pse_mc_inc_resp_t), pse_mc_inc }, {PSE_MC_SERVICE, PSE_MC_DEL, sizeof(pse_mc_del_req_t), sizeof(pse_mc_del_resp_t), pse_mc_del }, {PSE_TRUSTED_TIME_SERVICE, PSE_TIMER_READ, sizeof(pse_timer_read_req_t), sizeof(pse_timer_read_resp_t), pse_read_timer}, };

And pse_mc_create is our target handler.

5. VMC (Virtual Monotonic Counter?)

Now we are into create_vmc[6], described as “create a VMC in SQLite Database and return UUID of the VMC to the caller”. Huh, so the MC is essentially an entry in a SQLite DB. So how could a SQLite DB defend against rollback attacks given that it is just a normal file on the disk? Hang on. Let’s continue. When this is the first MC creation request ever happend, we need to create the VMC DB first, via initialize_sqlite_database_file[7]. This very first thing it does is to call read_rpdata[8]. As its file name implies, the SQLite DB accesses hardware (NV) to get the hash of the DB at certain epoch, which is essentailly a HW MC. We are getting close to the real HW…

6. PSDA_Service[9]

This file contains all functions used by the VMC SQLite DB to create/delete/update HW MC in the so-far-unknown HW via invoke_psda_service. An ephemeral session is established in the function to provide secure and authenticated communication with the mystery HW, through psda_invoke_service_ocall[10], and eventually psda_invoke_service[11].

7. Intel ME

Just looking at the psda_invoke_service is really insightful:

JVM_COMM_BUFFER commBuf; commBuf.TxBuf->buffer = psda_req_msg; commBuf.TxBuf->length = psda_req_msg_size; commBuf.RxBuf->buffer = psda_resp_msg; commBuf.RxBuf->length = psda_resp_msg_size; int response_code; ae_error_t ret; PROFILE_START("JHI_SendAndRecv2"); ret = PSDAService::instance().send_and_recv( PSDA_COMMAND_SERVICE, &commBuf, &response_code, NO_RETRY_ON_SESSION_LOSS); PROFILE_END("JHI_SendAndRecv2");

JVM? JHI? Finally, we have reached the dark side of the MC. Somehow some Java shit involves in communicating with the secret HW[13] — Yes, it is Intel ME! JHI stands for Java Host Interface, which is used to install Java applets into Dynamic Application Loader (DAL) firmware (inside ME) and communicate with them[12]. Yes, the SGX MC is essentially implemented in ME as a Java applet called “PSDA”!

8. PSDA.dalp

PSDAService::install_psda[14] installs the psda applet into ME using JHI. Not surprisingly, the source code for psda is not available. Instead, the download_prebuilt.sh script is used to download the applet binary directly, since it needs to signed by Intel — Right! Intel does not want anyone to install any applet into ME, for security reasons!!! After running the download script, PSDA.dalp will appear under psw/ae/data/prebuilt. It is encoded as Base64:

Copying out the Base64 encoded applet and running decoding, we have a chance to peek into the implementation of the applet:

com.intel.cryptocom.intel.langutilcom.intel.sgx.psdacom.intel.sgx.psda.commoncom.intel.sgx.psda.servicecom.intel.sgx.psda.service.base&com.intel.sgx.psda.service.entity.base)com.intel.sgx.psda.service.entity.command)com.intel.sgx.psda.service.entity.message)com.intel.sgx.psda.service.entity.storagecom.intel.sgx.psda.sessioncom.intel.sgx.psda.util com.intel.uicom.intel.util java.langPlatformServiceDalAppletBytesArrayInputStreamBytesArrayOutputStream CodeWrapper Constants#Constants$CSE_PLATFORM_CAPABILITIES!Constants$CSE_SERVICE_RESP_STATUS!Constants$EPHEMERAL_SESSION_STATEConstants$LT_SESSION_STATEConstants$MEDIAPATH_IDonstants$PSDA_COMMARPDataServiceServiceFactoryServiceMessageFactoryTrustedTimeServiceServiceHandlerPSDACommonMessage$PSDA_SERVICE_ID PSDAMessagePSDASerializableServiceCommandCSEUnknownReqResponsePRTCReadRequestPRTCReadResponse"ProtectedOutputSetStreamKeyRequest#ProtectedOutputSetStreamKeyResponseRPDataReadRequestRPDataResetRequestRPDataResponseRPDataUpdateRequestCapabilityRespMessageCertificateInfoRespMessageEphSessionMessageLTSessionMessagePlatformInfoReqMessageServiceMessagePSEPairingInfo SEFlashData EphSessionEphSessionManager LTSessionLTSessionManager CalendarUtil DALApiProxy DebugUtil FlashUtilSGXMonotonicCounterProxyCryptoException EpidAlgExHashAlgIllegalParameterExceptionNotSupportedExceptionOperationFailedExceptionRandom SigmaAlgExSymmetricBlockCipherAlgSymmetricSignatureAlg TypeConverterProtectedOutput UiExceptioniIllegalParameterExceptionUiIllegalUseExceptioCalendar DebugPrint FlashStorageFlashWearoutException IOException UtilExceptionimeExceptionObjectString IntelAppletSEMonotonicCounter StringBuilder Throwable StringBuffer API_VERSION AUDIO_OUTPUT!CAP_PROTECTED_OUTPUT_KEY_EXCHANGE CAP_RPDATACAP_TRUSTED_TIME EPID_GID_SIZEERROR_CAP_NOT_AVAILABLEERROR_INTERNALERROR_INVALID_PARAM%ERROR_PERSISTENT_DATA_WRITE_THROTTLEDERROR_UNKNOWN_REQFLASH_STRUCT_VERSIONHMAC_SHA256_SIZEINSTANCE_ID_LEINT_SIZELT_SESSION_STATE_ESTABLISHEDLT_SESSION_STATE_IN_PROGRESSLT_SESSION_STATE_NOT_EXIST MAC_KEY_SIZEMAX_SESSION_COUNTMC_EPOCH_LENGTH NONCE_SIZEPAYLOAD_IV_SIZEPAYLOAD_MAC_SIZEPRODUCT_TYPE_PSEPROTECTED_OUTPUT_STREAMKEY_SIZEPR_SIZEPSDA_BAD_PARAMETERPSDA_COMMAND_EPPSDA_COMMAND_LTPSDA_COMMAND_SERVICE PSDA_ILLEGAL_PARAMETER_EXCEPTIONPSDA_INTEGRITY_ERRORPSDA_INTERNAL_ERRORPSDA_INVALID_COMMANDPSDA_INVALID_SESSION_STATEPSDA_LT_PAIRING_NOT_EXISTPSDA_MSG_TYPE_CAP_QUERYPSDA_MSG_TYPE_CAP_RESULTPSDA_MSG_TYPE_CERT_INFO_QUERYPSDA_MSG_TYPE_CERT_INFO_RESULTPSDA_MSG_TYPE_EP_M1PSDA_MSG_TYPE_EP_M2PSDA_MSG_TYPE_EP_M3PSDA_MSG_TYPE_EP_M4PSDA_MSG_TYPE_LT_M1PSDA_MSG_TYPE_LT_M2PSDA_MSG_TYPE_LT_M3PSDA_MSG_TYPE_LT_M4PSDA_MSG_TYPE_SERV_REQPSDA_MSG_TYPE_SERV_RESPPSDA_NOT_PROVISIONEDPSDA_NOT_SUPPORTED_EXCEPTION$PSDA_PERSISTENT_DATA_WRITE_THROTTLEDPSDA_PLATFORM_KEYS_REVOKEDSDA_PROTOCOL_NOT_SUPPORTEDPSDA_SEQNO_CHECK_FAIL PSDA_SUCCESSPSDA_UNKNOWN_REQEUSTPSDA_UTIL_EXCEPTIONPSE_PROTECTED_OUTPUT_SERVICE PSE_PROTECTED_OUTPUT_SETSTEAMKEYPSE_RPDATA_READPSE_RPDATA_RESETPSE_RPDATA_SERVICEPSE_RPDATA_UPDATEPSE_TIMER_READPSE_TRUSED_TIME_SERVICEPS_COMMAND_INFO RPDATA_SIZESESSION_ACTIVESESSION_CLOSEDSESSION_IN_PROGRESSSESSION_KEY_SIZESGX_MONOTONIC_COUNTER_TYPE_RPMBSGX_MONOTONIC_COUNTER_TYPE_RPMCSUCCESSTIMER_EPOCH_LENGTHUNIX_TIMESTAMP_2015_01_01 VIDEO_OUTPUTbuffercapDescriptorVersiocertInfocodecount currentTime enableDebug encryptKeyencryptedPayload eph_sessionindeinstance lt_sessionlt_session_state_maskm_bIsFreshPairingm_bWritePending m_calendarm_idCsem_idPsm_macKeym_mk m_nonceCsem_pairing_infm_seqNum m_sessionKeym_sessionStatem_sigmaInstancem_sk mediaPathIdmkmsgBodymsgLenmsgTypemsgTypeExpRespSize pairing_info pairing_sk payloadIv payloadMac payloadSize prtc_epochpse_instance_ireservedrpEpochrp_datrp_epochrpdata rpdata_cur rpdata_newseqNumserialVersionUIDserviceCapDescriptor0serviceCapDescriptor1 set_time_infosgxMonotonicCounterskstatussw_instance_id timerEpochversionwrappedEncryptKeabandonSessionclearcreateServiceCommandcreateServiceMessage deriveIDseraseAllFlashData expandBufferfinalizeEphemeralSessionfinalizeLTSessionfindPairingInfogetCurrentEpidGidgetCurrentTimegetFlashDataSize invokeServiceisFreshPairingisSessionEstablishedonCloseonInitonInternalCommandonInternalOpenHandler printBuffeprintInt printreadFlashDatareadInreadLongonprocessLTSessioreadByte readBytes readObjectreadPairingCounter readShortresetresponseWithMessageseekset!setFlashStoragePropertyAntiReplay setResponse setSeqNumsetTimesigmaSetPropertyAppendSVNsizestartEphemeralSessionstartLongTermSession toByteArratoStringupdatePairingInfo updateRPData writeByte writeByteswriteFlashData_0writeFlashData_writeInt writeLong writeObject writeShortappend bytesToInt copyByteArraycreateortcocreateAlgcreateInstancedecryptCompletedeletedisposeenableHighUpdateFrequencyFlashencryptCofillByteArraygetEncryptedKeyRecord getMacKeySize getMessagegetRandomBytes getS1MessagegetS1MessageLengthgetS3MessageLength getSecretKeygetSecretKeySizegetSessionParametergetTime initialize isProvisionedlengthprocessCompleteprocessS2MessagesetIVsetKey setPropertysetResponseCode shortToBytes signCompleteverifyCompletewriteFlashData*+***-*D*)*;j)*L**)**Sd*+*(***+***%*%*&*&d*)*d**d**ddd*N*Sd*+**d*****dd**S*)dd*!**U*U*R*U*S*T*S*Uddddd*6*<*=*5*;dd*Tdd*L*O*Dd*@ddddddd

9. Intel ME Again

Although we only looked at the MC creation in SGX, this post has been long enough. Last thing about SGX MC is still about Intel ME. If you have seen dmesg like “Device doesn’t have valid ME Interface”, and neither /dev/mei nor /dev/mei0 is created, it is very likely that you are running a server platform with a different ME firmware (SPS) comparing to desktops/laptops[15], and the corresponding DAL firmware is not available in SPS. This means SGX MC is NOT available for servers!!! Well, I guess this does not really matter as people are in a rush to disable ME anyway… On the other hand, implementing SGX MC using ME seems to be the most reasonable choice (at least, we are still inside the CPU package), although this makes “SGX defends against ME attacks” statement in vain.

References:

[1] https://software.intel.com/en-us/node/709160

[2] https://github.com/01org/linux-sgx/blob/master/sdk/tae_service/tae_service.cpp

[3] https://github.com/01org/linux-sgx/blob/master/psw/uae_service/sgx_uae_service.cpp

[4] https://github.com/01org/linux-sgx/blob/master/psw/uae_service/uae_wrapper/src/tae_ocall_api.cpp

[5] https://github.com/01org/linux-sgx/blob/master/psw/ae/pse/pse_op/pse_op.cpp

[6] https://github.com/01org/linux-sgx/blob/master/psw/ae/pse/pse_op/monotonic_counter_database_sqlite_rpdb.cpp

[7] https://github.com/01org/linux-sgx/blob/master/psw/ae/pse/pse_op/monotonic_counter_database_sqlite_bin_hash_tree_utility.cpp

[8] https://github.com/01org/linux-sgx/blob/master/psw/ae/pse/pse_op/monotonic_counter_database_sqlite_access_hw_mc.cpp

[9] https://github.com/01org/linux-sgx/blob/master/psw/ae/pse/pse_op/psda_service.cpp

[10] https://github.com/01org/linux-sgx/blob/master/psw/ae/aesm_service/source/pse_op/pse_op_psda_ocall.cpp

[11] https://github.com/01org/linux-sgx/blob/master/psw/ae/aesm_service/source/pse_op/PSEClass.cpp

[12] https://github.com/intel/dynamic-application-loader-host-interface

[13] https://www.slideshare.net/codeblue_jp/igor-skochinsky-enpub

[14] https://github.com/01org/linux-sgx/blob/master/psw/ae/aesm_service/source/pse_op/PSDAService.cpp

[15] https://software.intel.com/en-us/forums/intel-business-client-software-development/topic/696357