

In this blog post, I'm discussing how to utilize Docker Secrets (a Docker Swarm service feature) to manage sensitive data (like password encryption keys, SSH private keys, SSL certificates etc.) for Dockerized application powered by IBM WebSphere Liberty Profile (WLP) application server. Docker Secrets helps to centrally manage these sensitive information while in rest or in transit (encrypted and securely transmitted to only those containers that need access to it and has explicit access to it). It is out of scope for this post to go deep into Docker secretes, however, if you need to familiarize yourself with Docker Secretes, refer to

Note: if you like to know how to program encryption/decryption within your Java application using passwordutilities-1.0 feature of WLP, see my blog

I'm going to write this post in a tutorial style, so that anyone interested to try can follow the steps.









Pre-requisite: In order to follow the steps outlined here, you have to have following: Good working knowledge of Docker Configured Docker Swarm environment (using Docker 1.13 or higher version) with at least one manager and one worker node or Docker Datacenter with Universal Control Plane (UCP) having manager node, worker node(s). It's good to have a separate Docker client node, so that you can remotely connect to manager and execute commands. Good working knowledge of IBM WebSphere Liberty Profile (https://developer.ibm.com/wasdev/blog/2013/03/29/introducing_the_liberty_profile/). Here is brief description, how we are going to utilize Docker Secretes with WLP. Password encryption key that is used to encrypt password for WLP KeyStore, TrustStore and any other password(s) used by WLP applications will be externalized and stored as Docker Secretes. Private key such as one stored in KeyStore (used to enable secure communication in WLP) will be externalized and stored as Docker Secretes. Here are some obvious benefits: Centrally manage all sensitive data. Since Docker enforces access control, only people with enough/right privilege(s) will have access to sensitive data. Only those container(s) and service(s) will have access to private/sensitive data which has explicit access as per need basis. Private information remains private while in rest or in transit. New Docker image created by 'docker commit' will not contain any sensitive data and also dump/package created by WLP server dump or package command, will not contain encryption key as it's externalized. See more insights about WLP password encryption here: https://www.ibm.com/support/knowledgecenter/en/SS7K4U_8.5.5/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/cwlp_pwd_encrypt.html and managing Docker Secrets here: https://docs.docker.com/engine/swarm/secrets/ Enough talk, now, let's start the real work. Below are the major steps that we'll carry out: Create Docker secrets for following that is being used by WLP: KeyStore

Truststore

Password Encryption key Build Docker image based on websphere-liberty:webProfile7 Create network Put together docker-compose.xml for deployment. Deploy application as Docker service.







Create Docker Secrets Here, we're going to use Docker

DOCKER_TLS_VERIFY

DOCKER_CERT_PATH

DOCKER_HOST If you are using Docker Datacenter, you can use GUI based UCP Admin Console to create the same. Note: label com.docker.ucp.access.label="<value> " is not mandatory unless you have access constraint defined. For detail refer to Authentication and authorization.

1) Create Docker Secrete with name keystore.jks, which basically is key database that stores private key to be used by WLP.



#Usage: docker secret create [OPTIONS] SECRET file|-

#Create a secret from a file or STDIN as content



$> docker secret create keystore.jks /mnt/nfs/dockershared/wlpapp/keystore.jks --label com.docker.ucp.access.label="dev"

idc9em1u3fki8k0z77ol91sh4 In this blog post, I'm discussing how to utilize Docker Secrets (a Docker Swarm service feature) to manage sensitive data (like password encryption keys, SSH private keys, SSL certificates etc.) for Dockerized application powered by IBM WebSphere Liberty Profile (WLP) application server. Docker Secrets helps to centrally manage these sensitive information while in rest or in transit (encrypted and securely transmitted to only those containers that need access to it and has explicit access to it). It is out of scope for this post to go deep into Docker secretes, however, if you need to familiarize yourself with Docker Secretes, refer to https://docs.docker.com/engine/swarm/secrets/ Note: if you like to know how to program encryption/decryption within your Java application using passwordutilities-1.0 feature of WLP, see my blog How to use WLP passwordUtilities feature for encryption/decryption I'm going to write this post in a tutorial style, so that anyone interested to try can follow the steps.Here, we're going to use Docker Commandline (CLI) and we'll execute Docker command from Docker client node remotely. You need have following three environment variables correctly setup in order to execute command remotely. Refer to https://docs.docker.com/engine/reference/commandline/cli/#description for detail.If you are using Docker Datacenter, you can use GUI based UCP Admin Console to create the same. Note: label com.docker.ucp.access.label=" 1) Create Docker Secrete with name keystore.jks, which basically is key database that stores private key to be used by WLP.

$> docker secret create truststore.jks /mnt/nfs/dockershared/wlpapp/truststore.jks --label com.docker.ucp.access.label="dev"

w8qs1o7pwrvl96nuamv97sb9t

$> docker secret create app_enc_key.xml /mnt/nfs/dockershared/wlpapp/app_enc_key.xml --label com.docker.ucp.access.label="dev"

kj3hcw4ss71hnudfgr6g32mxm

<server>

<variable name="wlp.password.encryption.key" value="#replaceMe#">

</variable>

</server>

$> docker secret ls

ID NAME CREATED UPDATED

idc9em1u3fki8k0z77ol91sh4 keystore.jks 3 hours ago 3 hours ago

kj3hcw4ss71hnudfgr6g32mxm app_enc_key.xml 21 seconds ago 21 seconds ago

w8qs1o7pwrvl96nuamv97sb9t truststore.jks 3 hours ago 3 hours ago









Building Docker Image:

$> cd /opt/ibm/wlp/bin

$> ./securityUtility encode #myStrongPassw0rd# --encoding=aes --key=#replaceMe#

{aes}AAj/El4TFm/8+9UFzWu5kCtURUiDIV/XKbGY/lT2SVKFij/+H38b11uhjh+Peo/rBA==



<server description="TestWLPApp">

<featuremanager>

<feature>javaee-7.0</feature>

<feature>localConnector-1.0</feature>

<feature>ejbLite-3.2</feature>

<feature>jaxrs-2.0</feature>

<feature>jpa-2.1</feature>

<feature>jsf-2.2</feature>

<feature>json-1.0</feature>

<feature>cdi-1.2</feature>

<feature>ssl-1.0</feature>

</featuremanager>

<include location="/run/secrets/app_enc_key.xml"/>

<httpendpoint host="*" httpport="9080" httpsport="9443" id="defaultHttpEndpoint"/>

<ssl clientauthenticationsupported="true" id="defaultSSLConfig" keystoreref="defaultKeyStore" truststoreref="defaultTrustStore"/>

<keystore id="defaultKeyStore" location="/run/secrets/keystore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/>

<keystore id="defaultTrustStore" location="/run/secrets/truststore.jks" password="{aes}ANGkm5cIca4hoPMh4EUeA4YYqVPAbo4HIqlB9zOCXp1n"/>

<applicationmonitor updatetrigger="mbean"/>

<datasource id="wlpappDS" jndiname="wlpappDS">

<jdbcdriver libraryref="OracleDBLib"/>

<properties.oracle password="{aes}AAj/El4TFm/8+9UFzWu5kCtURUiDIV/XKbGY/lT2SVKFij/+H38b11uhjh+Peo/rBA==" url="jdbc:oracle:thin:@192.168.xx.xxx:1752:WLPAPPDB" user="wlpappuser"/>

</datasource>

<library id="OracleDBLib">

<fileset dir="/apps/wlpapp/shared_lib" includes="ojdbc6-11.2.0.1.0.jar"/>

</library>

<webapplication contextRoot="wlpappctx" id="wlpapp" location="/apps/wlpapp/war/wlptest.war" name="wlpapp"/>

</server>

FROM websphere-liberty:webProfile7

COPY /mnt/nfs/dockershared/wlpapp/server.xml /opt/ibm/wlp/usr/servers/defaultServer/

RUN installUtility install --acceptLicense defaultServer

COPY /mnt/nfs/dockershared/wlpapp/wlptest.war /apps/wlpapp/war/

COPY /mnt/nfs/dockershared/wlpapp/ojdbc6-11.2.0.1.0.jar /apps/wlpapp/shared_lib/

CMD ["/opt/ibm/java/jre/bin/java","-javaagent:/opt/ibm/wlp/bin/tools/ws-javaagent.jar","-Djava.awt.headless=true","-jar","/opt/ibm/wlp/bin/tools/ws-server.jar","defaultServer"]



ssl-1.0

$> docker build -t 192.168.56.102/osboxes/wlptest:1.0 .

Sending build context to Docker daemon 56.9 MB

Step 1/7 : FROM websphere-liberty:webProfile7

---> c035090355f5

...

Step 4/7 : RUN installUtility install --acceptLicense defaultServer

---> Running in 2bce0d02e253

Checking for missing features required by the server ...

...

Successfully built 07fef794348e









Create Overlay network:

$> docker network create -d overlay --label com.docker.ucp.access.label="dev" --label com.docker.ucp.mesh.http=true my_hrm_network

naf8hvyx22n6lsvb4bq43z968



Put together docker-compose.yml

version: "3.1"

services:

wlpappsrv:

image: 192.168.56.102/osboxes/wlptest:1.0

volumes:

- /mnt/nfs/dockershared/wlpapp/server.xml:/opt/ibm/wlp/usr/servers/defaultServer/server.xml

networks:

- my_hrm_network

secrets:

- keystore.jks

- truststore.jks

- app_enc_key.xml

ports:

- 9080

- 9443

deploy:

placement:

constraints: [node.role == worker]

mode: replicated

replicas: 4

resources:

limits:

memory: 2048M

restart_policy:

condition: on-failure

max_attempts: 3

window: 6000s

labels:

- "com.docker.ucp.mesh.http.9080=external_route=http://mydockertest.com:8080,internal_port=9080"

- "com.docker.ucp.mesh.http.9443=external_route=sni://mydockertest.com:8443,internal_port=9443"

- "com.docker.ucp.access.label=dev"

networks:

my_hrm_network:

external:

name: my_hrm_network

secrets:

keystore.jks:

external: true

truststore.jks:

external: true

app_enc_key.xml:

external: true



Volume definition that maps server.xml in the container with the one in the NFS file system is optional. This mapping gives additional flexibility to update the server.xml. You can achieve similar or even better flexibility/portability by using Docker Swarm Config service. See my blog post - How to use Docker Swarm Configs service with WebSphere Liberty Profile for details. The secrets definition under service 'wlpappsrv' refers to the secrets definition in the root level, which in it turns refers to externally defined secret. "com.docker.ucp.mesh.http. " labels are totally optional and only required if you are using HRM. "com.docker.ucp.access.label" is also optional and required only if you have defined access constraints. Since, I'm using Swarm and HRM, I don't need to explicitly map the internal container ports to host port. If you need to map, you can use something like below for your port definition:

ports:

- 9080:9080

- 9443:9443 You may encounter situation that your container application is not able to access the secrets created under /run/secrets. It may be related to bug #31006. In order to resolve the issue use 'mode: 0444' while defining your secrets. Something like this:

secrets:

- source: keystore.jks

mode: 0444

...







Deploy the service

$> docker stack deploy --compose-file docker-compose.yml dev_WLPAPP

secrets Additional property secrets is not allowed

$> docker service ls

ID NAME MODE REPLICAS IMAGE

28xhhnbcnhfg dev_WLPAPP_wlpappsrv replicated 4/4 192.168.56.102/osboxes/wlptest:1.0



$> docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

2052806bbae3 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk01/dev_WLPAPP_wlpappsrv.3.m7apci6i1ks218ddnv4qsdbwv

541cf0f39b6e 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk01/dev_WLPAPP_wlpappsrv.4.wckec2jcjbmrhstftajh2zotr

ccdd7275fd7f 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk02/dev_WLPAPP_wlpappsrv.2.oke0fz2sifs5ej0vy63250wo9

7d5668a4d851 192.168.56.102/osboxes/wlptest:1.0 "/opt/ibm/java/jre..." 3 minutes ago Up 3 minutes 9080/tcp, 9443/tcp centosddcwrk02/dev_WLPAPP_wlpappsrv.1.r9gi0qllnh8r9u8popqg5mg5b



********************************************************************************

product = WebSphere Application Server 17.0.0.2 (wlp-1.0.17.cl170220170523-1818)

wlp.install.dir = /opt/ibm/wlp/

server.output.dir = /opt/ibm/wlp/output/defaultServer/

java.home = /opt/ibm/java/jre

java.version = 1.8.0

java.runtime = Java(TM) SE Runtime Environment (pxa6480sr4fp7-20170627_02 (SR4 FP7))

os = Linux (3.10.0-514.el7.x86_64; amd64) (en_US)

process = 1@e086b8c54a8d

********************************************************************************

[7/24/17 19:44:29:275 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager A CWWKE0001I: The server defaultServer has been launched.

...

[7/24/17 19:44:30:533 UTC] 00000017 com.ibm.ws.config.xml.internal.XMLConfigParser A CWWKG0028A: Processing included configuration resource: /run/secrets/app_enc_key.xml

[7/24/17 19:44:31:680 UTC] 00000001 com.ibm.ws.kernel.launch.internal.FrameworkManager I CWWKE0002I: The kernel started after 2.763 seconds

[7/24/17 19:44:31:990 UTC] 0000001f com.ibm.ws.kernel.feature.internal.FeatureManager I CWWKF0007I: Feature update started.

[7/24/17 19:44:45:877 UTC] 00000017 com.ibm.ws.security.ready.internal.SecurityReadyServiceImpl I CWWKS0007I: The security service is starting...

[7/24/17 19:44:47:262 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyInfoManager I CWWKS4103I: Creating the LTPA keys. This may take a few seconds.

[7/24/17 19:44:47:295 UTC] 00000017 ibm.ws.security.authentication.internal.jaas.JAASServiceImpl I CWWKS1123I: The collective authentication plugin with class name NullCollectiveAuthenticationPlugin has been activated.

[7/24/17 19:44:48:339 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyInfoManager A CWWKS4104A: LTPA keys created in 1.065 seconds. LTPA key file: /opt/ibm/wlp/output/defaultServer/resources/security/ltpa.keys

[7/24/17 19:44:48:365 UTC] 00000028 com.ibm.ws.security.token.ltpa.internal.LTPAKeyCreateTask I CWWKS4105I: LTPA configuration is ready after 1.107 seconds.

[7/24/17 19:44:57:514 UTC] 00000017 com.ibm.ws.app.manager.internal.monitor.DropinMonitor A CWWKZ0058I: Monitoring dropins for applications.

[7/24/17 19:44:57:651 UTC] 0000003f com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel defaultHttpEndpoint has been started and is now listening for requests on host * (IPv6) port 9080.

[7/24/17 19:44:57:675 UTC] 0000003f com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel defaultHttpEndpoint-ssl has been started and is now listening for requests on host * (IPv6) port 9443 .

[7/24/17 19:44:57:947 UTC] 00000017 com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel wasJmsEndpoint302 has been started and is now listening for requests on host localhost (IPv4: 127.0.0.1) port 7276.

[7/24/17 19:44:57:951 UTC] 00000017 com.ibm.ws.tcpchannel.internal.TCPChannel I CWWKO0219I: TCP Channel wasJmsEndpoint302-ssl has been started and is now listening for requests on host localhost (IPv4: 127.0.0.1) port 7286.

...



blue

/run/secrets/app_enc_key.xml

defaultHttpEndpoint-ssl

/run/secrets/keystore.jks

/run/secrets/truststore.jks

/run/secrets/app_enc_key.xml

https://mydockertest.com:8443/wlpappctx

https://<docker-container-host>:9443/<application-context>







Example using Load-Balancer



If you have a load-balancer in front and want to set-up a pass-through SSL, you can use SNI: aka SSL routing. Below is simple example using ha-proxy. You can also refer to HA-Proxy documentation here for details.

Here is haproxy.cfg for our example PoC:

# /etc/haproxy/haproxy.cfg, version 1.7

global

maxconn 4096



defaults

timeout connect 5000ms

timeout client 50000ms

timeout server 50000ms



frontend frontend_ssl_tcp

bind *:8443

mode tcp

tcp-request inspect-delay 5s

tcp-request content accept if { req_ssl_hello_type 1 }

default_backend bckend_ssl_default



backend bckend_ssl_default

mode tcp

balance roundrobin

server worker1 192.168.56.103:8443 check

server worker2 192.168.56.104:8443 check



Here is a Dockerfile for custom image:

FROM haproxy:1.7

COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

Build the image:

Note: execute the 'docker build ...' command from the same directory where Dockerbuild file is located.





$> docker build -t my_haproxy:1.7 .

2) Following command creates secret called truststore.jks using physical Java keystore file which contains trust certificatesFinally create the Docker secret call app_enc_key.xml, which basically refers to the fragment of xml wich contains definintion of password encryption keyNote: Docker secrets are available under '/run/secrets/' at runtime to any container which has explicit access to that secret.Here is how the /mnt/nfs/dockershared/wlpapp/app_enc_key.xml look like:Note: Make sure to replace the string '#replaceMe#' with your own password encryption key.Let's check and make sure all our secrets are properly created and listed:Now, let's first encrypt our keystore and trusstore passwords using the pre-defined encryption key and put together the server.xml for WLP server. We are going to use securityUtility tool that ships with IBM WLP to encrypt our password.Note: make sure your password encryption key matches to the one that is defined by 'wlp.password.encryption.key' property in app_enc_key.xml.Here I'm encoding my example password '#myStrongPassw0rd#' using encryption key '#replaceMe#' with encoding option ' aes '.Please note that encoding option ' xor ' ignores the encryption key and uses default.Now, we have our Docker secrets created and we have encrypted our password. It's time to put together our server.xml for WLP application server and build the Docker image. Here is how my server.xml looke like.As you can see, the location of defaultKeyStore, defaultTrustStore, and app_enc_key.xml is pointing to directory ''. It is, as mentioned before, because all private data created by Docker Secrets will be available for the assigned services under '' of the corresponding container.Now let's put together Dockerfile.Note: above, I'm copying my server.xml into /opt/ibm/wlp/usr/servers/defaultServer/ before running the installUtility as I'm adding few features required by my application including,Finally, we're going to build the Docker image.Note: 192.168.56.102 is my local Docker Trusted Registry (DTR).Once, the image is successfully built, make sure the image is available on all nodes of Docker Swarm. I'm not going show details how you distribute the image.> If you are using DTR, You can first push the image to the registry (using 'docker push ...', then connect to Docker Swarm host and execute 'docker pull ...'),> Other option is to use 'docker save ...' to save the image as tar file then load the image into Swarm using 'docker load ...'.Here, I'm deploying this into Docker Datacenter which has two UCP worker nodes, one UCP manager node and DTR node. I'm also going to use the HTTP routing mesh (HRM) , and User defined Overlay networks in swarm mode Note: User defined Docker network and HRM are NOT necessary to utilize the Docker secrets.Note: Label 'com.docker.ucp.mesh.http=true' is required while creating network in order to utilize HRM.Here is my compose file. Your may look different.Few notes about the docker-compose.ymlHere I'm using "docker stack deploy..." to deploy the service:Note: In certain cases, you may get "", error message. In order to resolve, make sure your compose file version to 3.1. In my case, where it's working fine, I've Docker version 17.03.2-ee4, API version: 1.27, Docker-Compose version 1.14.0.Once the service is deployed. You can list it using 'docker service ls ..." commandAnd list the replicated containers:And here is what the WLP messages.log shows (taken from one of the containers log file):As you can see (messages in), it's able to include the configuration fromand it also shows thathas been started and listening on port 9443; meaning that it's able to successfully load and open theandfiles using the encrypted password with encryption key defined inNow, it's time to access the application. In my case, since, I'm using HRM, I will access it as:If you are not using HRM; you may access it using: