TMWL Dec’19 — Scala 3 overview, Scala hacks and DevOps hints

Programmers say what they’ve learned

Since 2018, every month we’ve been sharing what we’ve learned in our commercial projects, oss projects, pet-projects and when attending dev community events.

The series — This Month We’ve Learned (at SoftwareMill) — encouraged us to be more curious and later on proud of our achievements, as we had learnt on the way that the tips we share are helpful and appreciated.

Let’s find out what we’ve learned in December 2019: Scala 3 overview, Scala hacks and devops’ hints!

Programmers who share their discoveries this time are:

“A tour of Scala 3” — best features of the upcoming language incarnation by Marcin

Considering the fact that Dotty has been in development for a couple of years now, we can safely say that the official release of Scala 3.0 is just around the corner — it is planned for late 2020! Martin Odersky, the creator of Scala language, has made a great recap of the upcoming version’s best features, including both his favourite, and these indicated by the community.

Let me point out my favourite three characteristics (described in depth by Martin in his talk):

Enums , the right way. enum is now going to be an official keyword in the language — no need to extend Enumeration anymore or use 3rd party libraries (like enumeratum). Enums can also take parameters, so it’s very easy to use them as ADTs, as well as GADTs.

, the right way. is now going to be an official keyword in the language — no need to extend anymore or use 3rd party libraries (like enumeratum). Enums can also take parameters, so it’s very easy to use them as ADTs, as well as GADTs. Union types , which I got very used to in other languages, that I use on a daily basis (like TypeScript).

, which I got very used to in other languages, that I use on a daily basis (like TypeScript). New way of defining typeclasses, via delegates. This novel syntax looks much more declarative & clean and should be approachable for beginners. In my opinion, typeclasses based on implicits in Scala 2.x carried lots of “noise”, were cumbersome and difficult to explain. Scala 3 aims to simplify this.

For more, I really recommend watching Martin’s talk. You can also try these features in your browser!

Scala goodies by Kasper

Before I started my journey with Scala I used to write code in Kotlin, which briefly speaking is a more concise and developer-friendly version of Java. One of the features I like in Kotlin is that it allows you to easily apply next functions on a current value while typing with the flow. To understand it better, let’s consider this simple example:

(For the purpose of this article I will be using let function, but there are also others)

The above code is equivalent to:

For most of the time when you write such a code you don’t think backwards from the ready-made solution. You think in a natural way — from what you already have, to what you still need to do with it in order to achieve your goals. This is where the let function shines. Of course you can extract each transformation to its own variable and eventually you will have the same feeling with a top to bottom processing. But extracting to a variable implies naming this variable, which as we know is the hardest problem in CS and sometimes, especially while prototyping, we would like to avoid that.

Once we’ve agreed that it is a useful thing, let’s see how it can be implemented in Scala.

In Kotlin the let function is built into the standard library. Obviously there is no such thing in Scala, or not that I am aware of, but we can easily add one ourselves by using an implicit class.

Ok, so that wasn’t hard, but can we do better?.

If you know cats library you should be familiar with the Functor typeclass. It applies given function to a value wrapped with some context, where context is for example Option , List , Future or Try . But here we only have a pure value without any context, so how can we use it? It turns out that no context is also a context called Id (from identity). Or in other words the value by itself forms an identity context. Once we know that, it is enough for use to import cats.Id (which comes with all instances of supported typeclasses) and cast our value to Id[T] like in the following example:

val original :Id[(BigDecimal, String)] = BigDecimal(1) -> “abc”

The cast is required to distinguish ambiguity.

The functor solution suffers from the need of upcasting, so its practical usage is fairly limited.

Despite this I still think that this is a great example of how a very concrete solution in Kotlin, is just a special case of more general solution in Scala.

CI server in Google Cloud Platform with the use of BigQuery by Grzegorz

One of our customers runs his CI server in Google Cloud Platform. The CI performs several integration tests and the results are stored in BigQuery. It allows us to perform several analyses on it — which tests fail the most, if there are any flaky tests, etc. Generally speaking — this setup is very good.

Technically — the CI produces a json file which is loaded to the BigQuery table with bq command line tool.

Few days ago the service account used by CI to load the data to BQ for some reason — which is not relevant here — got access to another GCP project. Nothing special. But this destroyed the whole integration test pipeline. Why? Because the bq tool — when ran for the very first time — initialises the setup. And when it has access to more than one project, it asks the user to choose one! The consequences are easy to predict — none of the IT steps was finished successfully and was put on hold to wait “forever” for the user’s input.

The first naive approach was to add the — project-id switch to the bq load command. But it didn’t help. Why? Because the question about the project was not asked when loading the data, but before, when initializing the bq setup.

I started to investigate it deeper. The first step was to enter the container when the test was performed and try to run the bq tool manually. And guess what? Everything just worked! Without asking about the project — even when the CI step was still hanging and hoping for the answer. This was really confusing…

So I prepared a simple lab: service account with access to multiple projects and fresh docker container without GCP setup but with google SDK installed.

The simple bq ls and voila! It asks about the project. So I chose one. The .bigqueryrc file was created and consecutive calls ran without any problems.

The next try: fresh container again, but instead of choosing the project I hit Ctrl-C to break the command. The .bigqueryrc file wasn’t created but.. consecutive calls didn’t ask about choosing the project any more! So there must be another change..

Indeed. After some time (longer than I’d wish) I found out it creates a file .config/gcloud/legacy_credentials/service-account-name@project-name.iam.gserviceaccount.com/singlestore_bq.json, and the file is always created, regardless it finishes the setup or not. And if this file exists — the bq assumes it’s initialized and doesn’t ask for anything. This also explained why the manual runs of bq succeeded on CI — the magic singlestore_bq.json was already created.

So how to workaround it? The solution was super easy. If the interactive setup is performed only if the .bigqueryrc didn’t exist — let’s create one! But what should be the content of this file? It turned out it’s not important. Simple touch .bigqueryrc did the trick.

I see several problems here. First — a lack of documentation. I spent several hours trying to figure out what’s going on there, but was not able to find a similar case in the docs. But maybe it’s only a lack in my googling skills?

Second: a lack of option which allows to set the project noninteractively. Looks like a bug in bq?

And last but not least — no option to debug what the bq tool is doing and why. All my findings were because of my obstinacy and a bit of luck.