Upon transition to distributed systems with a large number of instances of services to the utmost, there are problems of their discovery and load balancing between them. As a rule, for solving these problems specific solutions are used, like a Consul (https://www.consul.io/), Eureka (https://github.com/Netflix/eureka) or good old Zookeeper (https://zookeeper.apache.org/), with Nginx, HAProxy and some bridge between them (see registrator (https://github.com/gliderlabs/registrator)).

The main problem with this approach is a huge number of different integrations, and, as a consequence, failure points where something may go wrong. Because in addition to the above solutions, surely it will be used local small (or not small) PaaS (for example Mesosphere Marathon (https://mesosphere.github.io/marathon/) or Kubernetes). The latter, by the way, already store the necessary information about the environment (because all deployment goes through them). And we should ask a question, could we not use these separated solutions for service discovery and reuse Marathon or another orchestrator for solving this problem?

The short answer "Yes, we can."

Disposition

Ok, try to look what we have:

Apache Mesos and its faithful framework Marathon that is used for services orchestration (deployment, scaling, etc.).

Some services that are written with Spring Boot framework and its extension Spring Cloud.

Mesos without sugar (read as without frameworks) is a cluster resource management system, which could be extended by frameworks. Frameworks solve different problems. Some of them can launch short-lived tasks (Chronos) for batching and processing data for example. Other launch long-lived tasks (Marathon), or services that are processed requests for a while. And even more, appropriate frameworks for Hadoop or Jenkins exist.

Mesosphere Marathon is the very same framework, which could launch, stop, restart, scale and other things that are required to manage long-lived tasks or services.

Spring Cloud (http://projects.spring.io/spring-cloud/) is framework too, but for developing these services. It has an implementation of basic patterns for distributed systems and specific integrations with different service registries or configuration management systems like Consul or other that we've mentioned yet.

In Spring Cloud there are two separate implementations for service discovery problem-solving.

First, Netflix Zuul (https://github.com/Netflix/zuul) implements Server-Side Service Discovery (http://microservices.io/patterns/server-side-discovery.html) pattern. Its main idea is that several smart routers have information about services and their locations, different meta information about instances. These routers provide a limited number of static HTTP-resources that are used as proxies by clients. If we don't consider Spring, then we may say Nginx is a classic router on condition of its dynamic configuration. The pattern is presented as an image below.

The second implementation is called Client-Side Service Discovery (http://microservices.io/patterns/client-side-discovery.html). Its main difference from previous is there is no any router and no any additional failure point. Instead of a smart router, smart client load balancer is used. It is smart because it has information about balanced services that are needed to be called and it uses statistics for decision-making. Spring Cloud has the load balancer that is called Netflix Ribbon (https://github.com/Netflix/ribbon). And the pattern is presented as an image below.

In this series of articles, we will talk about client-side variation mostly, but we will touch server-side too.

@EnableDiscoveryClient

In Spring all the things (or almost all) start working with one or several dozens of annotations over classes or methods or variables. Also, some configuration in YAML files might be required, or some environment parameters.

From the reference, we know that magic-annotation @EnableDiscoveryClient over the main class of application enables (after configuration modification and starter inclusion) service discovery features. At least locally inside of our application. Easy peasy, even you don't have to strain:

@SpringBootApplication @EnableDiscoveryClient public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } }

After that, the magic without sorcery is happening. Spring find out this annotation and loads all configurations that are described in META-INF/spring.factories as loadable for EnableDiscoveryClient :

org.springframework.cloud.client.discovery.EnableDiscoveryClient=\ org.springframework.cloud.xxx.discovery.XXXDiscoveryClientConfiguration

What configurations could be loaded at all? And where their location could be found? For an answer to this question, we need to consider the fact that Spring Cloud consists of the base part and connectors from Spring Boot starters. The base part has an implementation of common patterns, have common beans and typical configuration. Connectors, in the opposite, have a particular implementation of concrete 3d-party solutions.

Let us say, if starter for Netflix Eureka is added to dependencies, then in classpath there will be one configuration factory. In the case of starter for Consul, there will be another.

But there is a sad thing. Annоtation by itself, in the case of production, not hello-world-development, is unusable, because right properties should be written in a proper place: bootstrap.yml . And each connector has its own "right. "

It is common cause all of the service registries, and Eureka, and Consul, and Marathon have different features and have different internals. At least different kinds of connection to them, API, specific features like DNS-discovery. Universal configuration is not possible, or at least very hard to achieve. And moreover, it is not necessary actually.

One step back to the configuration that is enabled by @EnableDiscoveryClient . And first that you might google, or find out through search in your favorite IDE, is the implementation of DiscoverClient interface. The primary (on the face of it by the way) interface looks like:

public interface DiscoveryClient { public String description(); public ServiceInstance getLocalServiceInstance(); public List<ServiceInstance> getInstances(String serviceId); public List<String> getServices(); }

Everything is pretty obvious. We could get a description for HealthIndicator . We could get ourselves. We could fetch instances of service with a particular identifier. And, at last, we could receive all registered services (more precise its identifiers).

It's time to implement an interface for fetching data from Marathon.

First blood

How to bring the data? It is a first problem that we need to solve. And what well it is not hard.

First, it has powerful API (https://mesosphere.github.io/marathon/api-console/index.html). And second, there is Java SDK (https://github.com/mesosphere/marathon-client).

Let's implement fetching services ids:

@Override public List<String> getServices() { try { return client.getApps() .getApps() //more apps for god of apps .parallelStream() .map(App::getId) //fetch identifiers .map(ServiceIdConverter::convertToServiceId) //some magic ;) .collect(Collectors.toList()); } catch (MarathonException e) { return Collections.emptyList(); } }

No magic except ServiceIdConverter::convertToServiceId . What a strange converter?! We need to dive deeper to the internal representation of service identifiers in Marathon. In general, they have following pattern: /group/path/app but a symbol / cannot be used as a part of a virtual host, because of HTTP specification. And, in this way, some parts of Spring Cloud, where service identifier is used as a virtual host, will not work. So instead of / , we will use a separator that is allowed to be in a hostname. Yes, you are right. It is a point. And we need a mapping between these two representations: /group/path/app and group.path.app . And magic converter does this job.

Fetching instances by service id is also not so hard:

@Override public List<ServiceInstance> getInstances(String serviceId) { try { return client.getAppTasks(ServiceIdConverter.convertToMarathonId(serviceId)) .getTasks() .parallelStream() .filter(task -> null == task.getHealthCheckResults() || task.getHealthCheckResults().stream().allMatch(HealthCheckResult::isAlive) ).map(task -> new DefaultServiceInstance( ServiceIdConverter.convertToServiceId(task.getAppId()), task.getHost(), task.getPorts().stream().findFirst().orElse(0), false )).collect(Collectors.toList()); } catch (MarathonException e) { log.error(e.getMessage(), e); return Collections.emptyList(); } }

The main thing that is needed to check is all service's health checks are passed: HealthCheckResult::isAlive , because we want to work only with health instances. Health checking feature is provided by Marathon itself. It has settings for setting up health checks, and it checks them with some interval. All that information could be fetched from API.

In addition to this, we shouldn't remember of converting service identifier inaccurate Marathon representation and vice versa. Also, we should choose only one port (basically first):

task.getPorts().stream().findFirst().orElse(0).

Wait, wait, wait - you might be said. What we go to do if service has several ports? Unfortunately, we have limited number of variants. On the one hand, we should return an object that implements ServiceInstance interface that has getPortmethod which returns only one port as you may guess. On the contrary, we don't know which ports from a list are used. Marathon doesn't give any information about it, so we simply take port that is defined first. Поэтому берём тот, что указан первым. Maybe luck.

This problem may be solved in a registrator-like way. The solution is to use any port in service identifier like that: group.path.app.8080 in the case of multiple ports.

We are little distracted. It's time to define our implementation as a bean:

@Configuration @ConditionalOnMarathonEnabled @ConditionalOnProperty(value = "spring.cloud.marathon.discovery.enabled", matchIfMissing = true) @EnableConfigurationProperties public class MarathonDiscoveryClientAutoConfiguration { @Autowired private Marathon marathonClient; @Bean public MarathonDiscoveryProperties marathonDiscoveryProperties() { return new MarathonDiscoveryProperties(); } @Bean @ConditionalOnMissingBean public MarathonDiscoveryClient marathonDiscoveryClient(MarathonDiscoveryProperties discoveryProperties) { MarathonDiscoveryClient discoveryClient = new MarathonDiscoveryClient(marathonClient, marathonDiscoveryProperties()); return discoveryClient; } }

What's important here. First, we use conditional annotations: @ConditionalOnMarathonEnabled and @ConditionalOnProperty . So if the feature is turned off through spring.cloud.marathon.discovery.enabled property, then configuration would not be loaded. Second, the annotation @ConditionalOnMissingBean is placed under the client bean that gives the end user an opportunity to replace client bean in its app.

It remains for us to do just a little. Let's configure Marathon client. Naive, but working, the implementation looks that:

spring: cloud: marathon: scheme: http #url scheme host: marathon #marathon host port: 8080 #marathon port

For reading these properties, we need configuration properties bean:

@ConfigurationProperties("spring.cloud.marathon") @Data //lombok is here public class MarathonProperties { @NotNull private String scheme = "http"; @NotNull private String host = "localhost"; @NotNull private int port = 8080; private String endpoint = null; public String getEndpoint() { if (null != endpoint) { return endpoint; } return this.getScheme() + "://" + this.getHost() + ":" + this.getPort(); } }

and configuration that is very similar to the previous configuration:

@Configuration @EnableConfigurationProperties @ConditionalOnMarathonEnabled public class MarathonAutoConfiguration { @Bean @ConditionalOnMissingBean public MarathonProperties marathonProperties() { return new MarathonProperties(); } @Bean @ConditionalOnMissingBean public Marathon marathonClient(MarathonProperties properties) { return MarathonClient.getInstance(properties.getEndpoint()); } }

After that we might go to our app and autowire DiscoveryClient bean:

@Autowired private DiscoveryClient discoveryClient; and, for example, fetch a list of instances for individual service: @RequestMapping("/instances") public List<ServiceInstance> instances() { return discoveryClient.getInstances("someservice"); }

But at this moment we have the first surprise. In the real world, our goal is not to fetch instances because we want to fetch them. We want to call them and load balancing our calls between instances. Sad but true, DiscoveryClient is not using for load balancing, at least implicitly. Ok, I know that it is used for dynamic routes registration in edge server implementation by Zuul, and it is used in actuator's health check indicators, but it's not so much right?

Conclusion

We were able to integrate with Marathon. It's cool. Even more, we can even now get a list of services and their instances.

But we have at least two unsolved problems. First, it is our configuration contains only one instance of Marathon. If it fails, we wouldn't have information. No information, no right decisions. And second. We are not able to load balancing at the moment without some additional explicit programming in every app that we develop. So actually we have a toy, not useful and production-ready solution.

See you in next part