uPickle (pronounced micro-pickle) is a lightweight JSON and binary (MessagePack) serialization library for Scala. It's key features are:

Simple to use, with nice human-readable JSON output

Very high Performance; faster than Play-Json, Circe, or Argonaut by a large margin

Simple & easy to understand JSON Processing API, that should be instantly familiar to anyone whose processed JSON in Python, Ruby, or Javascript

Flexible and easily customizable

Zero dependencies; can be included in any project without worrying about conflicts

ScalaJS support, allowing transfer of structured data between the JVM and Javascript

Supports binary serialization via the MessagePack format, for even faster serialization/de-serialization and smaller payloads.

This documentation is meant as a thorough reference to this library. For a hands-on introduction, take a look at the following blog post:

Getting Started



"com.lihaoyi" %% "upickle" % "0.9.5" // SBT ivy"com.lihaoyi::upickle:0.9.5" // Mill

And then you can immediately start writing and reading common Scala objects to strings:

import upickle.default._ write(1) ==> "1" write(Seq(1, 2, 3)) ==> "[1,2,3]" read[Seq[Int]]("[1,2,3]") ==> List(1, 2, 3) write((1, "omg", true)) ==> """[1,"omg",true]""" read[(Int, String, Boolean)]("""[1,"omg",true]""") ==> (1, "omg", true)

Or to compact byte arrays, using the MessagePack format:

import upickle.default._ writeBinary(1) ==> Array(1) writeBinary(Seq(1, 2, 3)) ==> Array(0x93.toByte, 1, 2, 3) readBinary[Seq[Int]](Array[Byte](0x93.toByte, 1, 2, 3)) ==> List(1, 2, 3) val serializedTuple = Array[Byte](0x93.toByte, 1, 0xa3.toByte, 111, 109, 103, 0xc3.toByte) writeBinary((1, "omg", true)) ==> serializedTuple readBinary[(Int, String, Boolean)](serializedTuple) ==> (1, "omg", true)

ScalaJS

For ScalaJS applications, use this dependencies instead:

"com.lihaoyi" %%% "upickle" % "0.9.5" // SBT ivy"com.lihaoyi::upickle::0.9.5" // Mill

Other than that, everything is used the same way. upickle-0.9.4 is only compatible with ScalaJS 0.6.x.

Scala 2.10

uPickle does not support Scala 2.10; only 2.11 and 2.12 are supported

Basics



Builtins

This is a non-comprehensive list of what the most commonly-used types pickle to using uPickle. To begin, let's import upickle

import upickle.default._

Booleans are serialized as JSON booleans

Numbers are serialized as JSON numbers

Except for Long s, which too large for Javascript. These are serialized as JSON Strings, keeping the interchange format compatible with the browser's own JSON parser, which provides the best performance in Scala.js

write(12: Long) ==> "12" write(4000000000000L: Long) ==> "4000000000000" // large longs are written as strings, to avoid floating point rounding write(9223372036854775807L: Long) ==> "\"9223372036854775807\""

Special values of Double s and Float s are also serialized as Strings

Both Char s and String s are serialized as Strings

Array s and most immutable collections are serialized as JSON lists

write(Array(1, 2, 3)) ==> "[1,2,3]" // You can pass in an `indent` parameter to format it nicely write(Array(1, 2, 3), indent = 4) ==> """[ | 1, | 2, | 3 |]""".stripMargin write(Seq(1, 2, 3)) ==> "[1,2,3]" write(Vector(1, 2, 3)) ==> "[1,2,3]" write(List(1, 2, 3)) ==> "[1,2,3]" import collection.immutable.SortedSet write(SortedSet(1, 2, 3)) ==> "[1,2,3]"

Option s are serialized as JSON lists with 0 or 1 element

Tuples of all sizes (1-22) are serialized as heterogenous JSON lists

Case classes of sizes 1-22 are serialized as JSON dictionaries with the keys being the names of each field. To begin with, you need to define a serializer in the Case Class's companion object:

import upickle.default.{ReadWriter => RW, macroRW}

After that, you can begin serializing that case class.

case class Thing(myFieldA: Int, myFieldB: String) object Thing{ implicit val rw: RW[Thing] = macroRW } case class Big(i: Int, b: Boolean, str: String, c: Char, t: Thing) object Big{ implicit val rw: RW[Big] = macroRW }

import upickle._ write(Thing(1, "gg")) ==> """{"myFieldA":1,"myFieldB":"gg"}""" read[Thing]("""{"myFieldA":1,"myFieldB":"gg"}""") ==> Thing(1, "gg") write(Big(1, true, "lol", 'Z', Thing(7, ""))) ==> """{"i":1,"b":true,"str":"lol","c":"Z","t":{"myFieldA":7,"myFieldB":""}}""" write(Big(1, true, "lol", 'Z', Thing(7, "")), indent = 4) ==> """{ | "i": 1, | "b": true, | "str": "lol", | "c": "Z", | "t": { | "myFieldA": 7, | "myFieldB": "" | } |}""".stripMargin

Sealed hierarchies are serialized as tagged values, the serialized object tagged with the full name of the instance's class:

sealed trait IntOrTuple object IntOrTuple{ implicit val rw: RW[IntOrTuple] = RW.merge(IntThing.rw, TupleThing.rw) } case class IntThing(i: Int) extends IntOrTuple object IntThing{ implicit val rw: RW[IntThing] = macroRW } case class TupleThing(name: String, t: (Int, Int)) extends IntOrTuple object TupleThing{ implicit val rw: RW[TupleThing] = macroRW }

write(IntThing(1)) ==> """{"$type":"upickle.example.Sealed.IntThing","i":1}""" write(TupleThing("naeem", (1, 2))) ==> """{"$type":"upickle.example.Sealed.TupleThing","name":"naeem","t":[1,2]}""" // You can read tagged value without knowing its // type in advance, just use type of the sealed trait read[IntOrTuple]("""{"$type":"upickle.example.Sealed.IntThing","i":1}""") ==> IntThing(1)

Serializability is recursive; you can serialize a type only if all its members are serializable. That means that collections, tuples and case-classes made only of serializable members are themselves serializable

case class Foo(i: Int) object Foo{ implicit val rw: RW[Foo] = macroRW } case class Bar(name: String, foos: Seq[Foo]) object Bar{ implicit val rw: RW[Bar] = macroRW }

Nulls serialize into JSON nulls, as you would expect

uPickle only throws exceptions on unpickling; if a pickler is properly defined, serializing a data structure to a String should never throw an exception.

All these examples can be similarly serialized to MessagePack-formatted binaries, in the same way: JSON booleans become MessagePack booleans, lists become MessagePack lists, and so on. Reading and writing MessagePack binary data is typically significantly faster than reading and writing JSON, and the serialized data is also significantly smaller.

Read/Writing Other Things

Apart from reading & writing java.lang.String s, allows you to easily read from alternate sources such as CharSequence s, Array[Byte] s, java.io.File s and java.nio.file.Path s:

import upickle.default._ val original = """{"myFieldA":1,"myFieldB":"gg"}""" read[Thing](original) ==> Thing(1, "gg") read[Thing](original: CharSequence) ==> Thing(1, "gg") read[Thing](original.getBytes) ==> Thing(1, "gg")

import upickle.default._ val original = """{"myFieldA":1,"myFieldB":"gg"}""" import java.nio.file.Files val f = Files.createTempFile("", "") Files.write(f, original.getBytes) read[Thing](f) ==> Thing(1, "gg") read[Thing](f.toFile) ==> Thing(1, "gg")

Reading from large files is automatically streamed so you do not read the entire file into memory. You can use writeTo to serialize your data to an arbitrary java.io.Writer / java.io.OutputStream : this can be streamed directly to files or over the network without having to accumulate the serialized JSON in memory.

Defaults

If a field is missing upon deserialization, uPickle uses the default value if one exists

If a field at serialization time has the same value as the default, uPickle leaves it out of the serialized blob

write(FooDefault(i = 11, s = "lol")) ==> """{"i":11}""" write(FooDefault(i = 10, s = "lol")) ==> """{}""" write(FooDefault()) ==> """{}"""

This allows you to make schema changes gradually, assuming you have already pickled some data and want to add new fields to the case classes you pickled. Simply give the new fields a default value (e.g. "" for Strings, or wrap it in an Option[T] and make the default None ) and uPickle will happily read the old data, filling in the missing field using the default value.

Supported Types

Out of the box, uPickle supports writing and reading the following types:

Boolean , Byte , Char , Short , Int , Long , Float , Double

, , , , , , , Tuple s from 1 to 22

s from 1 to 22 Immutable Seq , List , Vector , Set , SortedSet , Option , Array , Map s, and all other collections with a reasonable CanBuildFrom implementation

, , , , , , , s, and all other collections with a reasonable implementation Duration , Either

, Stand-alone case class es and case object s, and their generic equivalents,

es and s, and their generic equivalents, Non-generic case class es and case object s that are part of a sealed trait or sealed class hierarchy

es and s that are part of a or hierarchy sealed trait and sealed class es themselves, assuming that all subclasses are picklable

and es themselves, assuming that all subclasses are picklable UUID s

s null

Readability/writability is recursive: a container such as a Tuple or case class is only readable if all its contents are readable, and only writable if all its contents are writable. That means that you cannot serialize a List[Any] , since uPickle doesn't provide a generic way of serializing Any . Case classes are only serializable up to 22 fields.

Case classes are serialized using the apply and unapply methods on their companion objects. This means that you can make your own classes serializable by giving them companions apply and unapply . sealed hierarchies are serialized as tagged unions: whatever the serialization of the actual object, together with the fully-qualified name of its class, so the correct class in the sealed hierarchy can be reconstituted later.

That concludes the list of supported types. Anything else is not supported by default, but you can add support using Custom Picklers

Common Operations

The following common operations are available on any uPickle module, e.g. upickle.default or upickle.legacy :

trait Api extends upickle.core.Types with implicits.Readers with implicits.Writers with WebJson with Api.NoOpMappers with JsReadWriters with MsgReadWriters{ /** * Reads the given MessagePack input into a Scala value */ def readBinary[T: Reader](s: upack.Readable): T = s.transform(reader[T]) /** * Reads the given JSON input into a Scala value */ def read[T: Reader](s: ujson.Readable): T = s.transform(reader[T]) def reader[T: Reader] = implicitly[Reader[T]] /** * Write the given Scala value as a JSON string */ def write[T: Writer](t: T, indent: Int = -1, escapeUnicode: Boolean = false): String = { transform(t).to(ujson.StringRenderer(indent, escapeUnicode)).toString } /** * Write the given Scala value as a MessagePack binary */ def writeBinary[T: Writer](t: T): Array[Byte] = { transform(t).to(new upack.MsgPackWriter(new ByteArrayOutputStream())).toByteArray } /** * Write the given Scala value as a JSON struct */ def writeJs[T: Writer](t: T): ujson.Value = transform(t).to[ujson.Value] /** * Write the given Scala value as a MessagePack struct */ def writeMsg[T: Writer](t: T): upack.Msg = transform(t).to[upack.Msg] /** * Write the given Scala value as a JSON string to the given Writer */ def writeTo[T: Writer](t: T, out: java.io.Writer, indent: Int = -1, escapeUnicode: Boolean = false): Unit = { transform(t).to(new ujson.Renderer(out, indent = indent, escapeUnicode)) } /** * Write the given Scala value as a JSON string via a `geny.Writable` */ def stream[T: Writer](t: T, indent: Int = -1, escapeUnicode: Boolean = false): geny.Writable = new geny.Writable{ def writeBytesTo(out: java.io.OutputStream) = { val w = new java.io.OutputStreamWriter(out, java.nio.charset.StandardCharsets.UTF_8) transform(t).to(new ujson.BaseRenderer(w, indent = indent, escapeUnicode)) w.flush() } } /** * Write the given Scala value as a MessagePack binary to the given OutputStream */ def writeBinaryTo[T: Writer](t: T, out: java.io.OutputStream): Unit = { streamBinary[T](t).writeBytesTo(out) } /** * Write the given Scala value as a MessagePack binary via a `geny.Writable` */ def streamBinary[T: Writer](t: T): geny.Writable = new geny.Writable{ def writeBytesTo(out: java.io.OutputStream) = transform(t).to(new upack.MsgPackWriter(out)) } def writer[T: Writer] = implicitly[Writer[T]] def readwriter[T: ReadWriter] = implicitly[ReadWriter[T]] case class transform[T: Writer](t: T) extends upack.Readable with ujson.Readable { def transform[V](f: Visitor[_, V]): V = writer[T].transform(t, f) def to[V](f: Visitor[_, V]): V = transform(f) def to[V](implicit f: Reader[V]): V = transform(f) }

Customization



Custom Picklers

import upickle.default._ case class Wrap(i: Int) implicit val fooReadWrite: ReadWriter[Wrap] = readwriter[Int].bimap[Wrap](_.i, Wrap(_)) write(Seq(Wrap(1), Wrap(10), Wrap(100))) ==> "[1,10,100]" read[Seq[Wrap]]("[1,10,100]") ==> Seq(Wrap(1), Wrap(10), Wrap(100))

You can use the readwriter[T].bimap[V] function to create a pickler that reads/writes a type V , using the pickler for type T , by providing a conversion function between them.

The type you are .bimap ing to doesn't need to be a case class, or be pickleable in any way, as long as the type you are .bimap ing from is pickleable. The following example demonstrates using .bimap to define a serializer for non-case Scala class

class CustomThing2(val i: Int, val s: String) object CustomThing2 { implicit val rw = upickle.default.readwriter[String].bimap[CustomThing2]( x => x.i + " " + x.s, str => { val Array(i, s) = str.split(" ", 2) new CustomThing2(i.toInt, s) } ) }

Note that when writing custom picklers, it is entirely up to you to get it right, e.g. making sure that an object that gets round-trip pickled/unpickled comes out the same as when it started.

Lastly, if you want more control over exactly how something is serialized, you can use readwriter[Js.Value].bimap to give yourself access to the raw JSON AST:

import upickle.default._ case class Bar(i: Int, s: String) implicit val fooReadWrite: ReadWriter[Bar] = readwriter[ujson.Value].bimap[Bar]( x => ujson.Arr(x.s, x.i), json => new Bar(json(1).num.toInt, json(0).str) ) write(Bar(123, "abc")) ==> """["abc",123]""" read[Bar]("""["abc",123]""") ==> Bar(123, "abc")

Custom Keys

uPickle allows you to specify the key that a field is serialized with via a @key annotation

case class KeyBar(@upickle.implicits.key("hehehe") kekeke: Int) object KeyBar{ implicit val rw: RW[KeyBar] = macroRW }

Practically, this is useful if you want to rename the field within your Scala code while still maintaining backwards compatibility with previously-pickled objects. Simple rename the field and add a @key("...") with the old name so uPickle can continue to work with the old objects correctly.

You can also use @key to change the name used when pickling the case class itself. Normally case classes are pickled without their name, but an exception is made for members of sealed hierarchies which are tagged with their fully-qualified name. uPickle allows you to use @key to override what the class is tagged with:

sealed trait A object A{ implicit val rw: RW[A] = RW.merge(B.rw, macroRW[C.type]) } @upickle.implicits.key("Bee") case class B(i: Int) extends A object B{ implicit val rw: RW[B] = macroRW } case object C extends A

This is useful in cases where:

you wish to rename the class within your Scala code, or move it to a different package, but want to preserve backwards compatibility with previously pickled instances of that class

you try to tackle the resource issue (bandwidth, storage, CPU) because FQNs might get quite long

Custom Configuration

Often, there will be times that you want to customize something on a project-wide level. uPickle provides hooks in letting you subclass the upickle.Api trait to create your own bundles apart from the in-built upickle.default and upickle.legacy . The following example demonstrates how to customize a bundle to automatically snake_case all dictionary keys.

object SnakePickle extends upickle.AttributeTagged{ def camelToSnake(s: String) = { s.split("(?=[A-Z])", -1).map(_.toLowerCase).mkString("_") } def snakeToCamel(s: String) = { val res = s.split("_", -1).map(x => x(0).toUpper + x.drop(1)).mkString s(0).toLower + res.drop(1) } override def objectAttributeKeyReadMap(s: CharSequence) = snakeToCamel(s.toString) override def objectAttributeKeyWriteMap(s: CharSequence) = camelToSnake(s.toString) override def objectTypeKeyReadMap(s: CharSequence) = snakeToCamel(s.toString) override def objectTypeKeyWriteMap(s: CharSequence) = camelToSnake(s.toString) } // Default read-writing upickle.default.write(Thing(1, "gg")) ==> """{"myFieldA":1,"myFieldB":"gg"}""" upickle.default.read[Thing]("""{"myFieldA":1,"myFieldB":"gg"}""") ==> Thing(1, "gg") implicit def thingRW: SnakePickle.ReadWriter[Thing] = SnakePickle.macroRW // snake_case_keys read-writing SnakePickle.write(Thing(1, "gg")) ==> """{"my_field_a":1,"my_field_b":"gg"}""" SnakePickle.read[Thing]("""{"my_field_a":1,"my_field_b":"gg"}""") ==> Thing(1, "gg")

If you are using uPickle to convert JSON from another source into Scala data structures, you can also configure it to map Option[T] s to null s when the option is None :

object OptionPickler extends upickle.AttributeTagged { override implicit def OptionWriter[T: Writer]: Writer[Option[T]] = implicitly[Writer[T]].comap[Option[T]] { case None => null.asInstanceOf[T] case Some(x) => x } override implicit def OptionReader[T: Reader]: Reader[Option[T]] = { new Reader.Delegate[Any, Option[T]](implicitly[Reader[T]].map(Some(_))){ override def visitNull(index: Int) = None } } }

This custom configuration allows you to treat null s as None s and anything else as Some(...) s. Simply import OptionPickler._ instead of the normal uPickle import throughout your project and you'll have the customized reading/writing available to you.

You can also use a custom configuration to change how 64-bit Long s are handled. By default, small longs that can be represented exactly in 64-bit Double s are written as raw numbers, while larger values (n > 2^53) are written as strings. This is to ensure the values are not truncated when the serialized JSON is then manipulated, e.g. by Javascript which truncates all large numbers to Doubles. If you wish to always write Longs as Strings, or always write them as numbers (at risk of truncation), you can do so as follows:

upickle.default.write(123: Long) ==> "123" upickle.default.write(Long.MaxValue) ==> "\"9223372036854775807\"" object StringLongs extends upickle.AttributeTagged{ override implicit val LongWriter = new Writer[Long] { def write0[V](out: Visitor[_, V], v: Long) = out.visitString(v.toString, -1) } } StringLongs.write(123: Long) ==> "\"123\"" StringLongs.write(Long.MaxValue) ==> "\"9223372036854775807\"" object NumLongs extends upickle.AttributeTagged{ override implicit val LongWriter = new Writer[Long] { def write0[V](out: Visitor[_, V], v: Long) = out.visitFloat64String(v.toString, -1) } } NumLongs.write(123: Long) ==> "123" NumLongs.write(Long.MaxValue) ==> "9223372036854775807"

Limitations



uPickle doesn't currently support:

Circular object graphs

Reflective reading and writing

Read/writing of untyped values e.g. Any

Read/writing arbitrarily shaped objects

Read/writing case classes with multiple parameter lists.

Most of these limitations are inherent in the fact that ScalaJS does not support reflection, and are unlikely to ever go away. In general, uPickle by default can serialize statically-typed, tree-shaped, immutable data structures. Anything more complex requires Custom Picklers

Manual Sealed Trait Picklers

Due to a bug in the Scala compiler SI-7046, automatic sealed trait pickling can fail unpredictably. This can be worked around by instead using the macroRW and merge methods to manually specify which sub-types of a sealed trait to consider when pickling:

sealed trait TypedFoo object TypedFoo{ import upickle.default._ implicit val readWriter: ReadWriter[TypedFoo] = ReadWriter.merge( macroRW[Bar], macroRW[Baz], macroRW[Quz] ) case class Bar(i: Int) extends TypedFoo case class Baz(s: String) extends TypedFoo case class Quz(b: Boolean) extends TypedFoo }

uJson



uJson is uPickle's JSON library, which can be used to easily manipulate JSON source and data structures without converting them into Scala case-classes. This all lives in the ujson package. Unlike many other Scala JSON libraries that come with their own zoo of new concepts, abstractions, and techniques, uJson has a simple & predictable JSON API that should be instantly familiar to anyone coming from scripting languages like Ruby, Python or Javascript.

uJson comes bundled with uPickle, or can be used stand-alone via the following package coordinates:

libraryDependencies += "com.lihaoyi" %% "ujson" % "0.9.5"

Construction

You can use ujson to conveniently construct JSON blobs, either programmatically:

import ujson.Value val json0 = ujson.Arr( ujson.Obj("myFieldA" -> ujson.Num(1), "myFieldB" -> ujson.Str("g")), ujson.Obj("myFieldA" -> ujson.Num(2), "myFieldB" -> ujson.Str("k")) ) val json = ujson.Arr( // The `ujson.Num` and `ujson.Str` calls are optional ujson.Obj("myFieldA" -> 1, "myFieldB" -> "g"), ujson.Obj("myFieldA" -> 2, "myFieldB" -> "k") ) json0 ==> json

Or parsing them from strings, byte arrays or files:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]""" val json = ujson.read(str) json(0)("myFieldA").num ==> 1 json(0)("myFieldB").str ==> "g" json(1)("myFieldA").num ==> 2 json(1)("myFieldB").str ==> "k" ujson.write(json) ==> str

ujson.Js ASTs are mutable, and can be modified before being re-serialized to strings:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]""" val json: ujson.Value = ujson.read(str) json.arr.remove(1) json(0)("myFieldA") = 1337 json(0)("myFieldB") = json(0)("myFieldB").str + "lols"

You can also use the `_` shorthand syntax to update a JSON value in place, without having to duplicate the whole path:

val str = """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]""" val json: ujson.Value = ujson.read(str) json(0)("myFieldA") = _.num + 100 json(1)("myFieldB") = _.str + "lol"

case classes or other Scala data structures can be converted to ujson.Js ASTs using upickle.default.writeJs , and the ujson.Js ASTs can be converted back using upickle.default.readJs or plain upickle.default.read :

val data = Seq(Thing(1, "g"), Thing(2, "k")) val json = upickle.default.writeJs(data) json.arr.remove(1) json(0)("myFieldA") = 1337 upickle.default.read[Seq[Thing]](json) ==> Seq(Thing(1337, "g"))

JSON Utilities

uJson comes with some convenient utilities on the `ujson` package:

def transform[T](t: Readable, v: upickle.core.Visitor[_, T]) = t.transform(v) /** * Read the given JSON input as a JSON struct */ def read(s: Readable): Value.Value = transform(s, Value) def copy(t: Value.Value): Value.Value = transform(t, Value) /** * Write the given JSON struct as a JSON String */ def write(t: Value.Value, indent: Int = -1, escapeUnicode: Boolean = false): String = { transform(t, StringRenderer(indent, escapeUnicode)).toString } /** * Write the given JSON struct as a JSON String to the given Writer */ def writeTo(t: Value.Value, out: java.io.Writer, indent: Int = -1, escapeUnicode: Boolean = false): Unit = { transform(t, Renderer(out, indent, escapeUnicode)) } /** * Parse the given JSON input, failing if it is invalid */ def validate(s: Readable): Unit = transform(s, NoOpVisitor) /** * Parse the given JSON input and write it to a string with * the configured formatting */ def reformat(s: Readable, indent: Int = -1, escapeUnicode: Boolean = false): String = { transform(s, StringRenderer(indent, escapeUnicode)).toString } /** * Parse the given JSON input and write it to a string with * the configured formatting to the given Writer */ def reformatTo(s: Readable, out: java.io.Writer, indent: Int = -1, escapeUnicode: Boolean = false): Unit = { transform(s, Renderer(out, indent, escapeUnicode)).toString }

Transformations

uJson allows you seamlessly convert between any of the following forms you may find your JSON in:

Case classes & Scala data-types

ujson.Js ASTs

ASTs String s

s CharSequence s

s Array[Byte] s

s Third-party JSON ASTs (Argonaut, Circe, Json4s, Play-Json)

This is done using the ujson.transform(source, dest) function:

// It can be used for parsing JSON into an AST val exampleAst = ujson.Arr(1, 2, 3) ujson.transform("[1, 2, 3]", Value) ==> exampleAst // Rendering the AST to a string ujson.transform(exampleAst, StringRenderer()).toString ==> "[1,2,3]" // Or to a byte array ujson.transform(exampleAst, BytesRenderer()).toBytes ==> "[1,2,3]".getBytes // Re-formatting JSON, either compacting it ujson.transform("[1, 2, 3]", StringRenderer()).toString ==> "[1,2,3]" // or indenting it ujson.transform("[1, 2, 3]", StringRenderer(indent = 4)).toString ==> """[ | 1, | 2, | 3 |]""".stripMargin // `transform` takes any `Transformable`, including byte arrays and files ujson.transform("[1, 2, 3]".getBytes, StringRenderer()).toString ==> "[1,2,3]"

All transformations from A to B using ujson.transform happen in a direct fashion: there are no intermediate JSON ASTs being generated, and performance is generally very good.

You can use ujson.transform to validate JSON in a streaming fashion:

The normal upickle.default.read/write methods to serialize Scala data-types is just shorthand for ujson.transform, using a upickle.default.transform(foo) as the source or a upickle.default.reader[Foo] as the destination:

Other ASTs

uJson does not provide any other utilities are JSON that other libraries do: zippers, lenses, combinators, etc.. However, uJson can be used to seamlessly convert between the JSON AST of other libraries! This means if some other library provides a more convenient API for some kind of processing you need to do, you can easily parse to that library's AST, do whatever you need, and convert back after.

As mentioned earlier, conversions are fast and direct, and happen without creating intermediate JSON structures in the process. The following examples demonstrate how to use the conversion modules for Argonaut, Circe, Json4s, and Play Json.

Each example parses JSON from a string into that particular library's JSON AST, manipulates the AST using that library, un-pickles it into Scala data types, then serializes those data types first into that library's AST then back to a st.ring

Argonaut

libraryDependencies += "com.lihaoyi" %% "ujson-argonaut" % "0.9.5"

import ujson.argonaut.ArgonautJson val argJson: argonaut.Json = ArgonautJson( """["hello", "world"]""" ) val updatedArgJson = argJson.withArray(_.map(_.withString(_.toUpperCase))) val items: Seq[String] = ArgonautJson.transform( updatedArgJson, upickle.default.reader[Seq[String]] ) items ==> Seq("HELLO", "WORLD") val rewritten = upickle.default.transform(items).to(ArgonautJson) val stringified = ArgonautJson.transform(rewritten, StringRenderer()).toString stringified ==> """["HELLO","WORLD"]"""

Circe

libraryDependencies += "com.lihaoyi" %% "ujson-circe" % "0.9.5"

import ujson.circe.CirceJson val circeJson: io.circe.Json = CirceJson( """["hello", "world"]""" ) val updatedCirceJson = circeJson.mapArray(_.map(x => x.mapString(_.toUpperCase))) val items: Seq[String] = CirceJson.transform( updatedCirceJson, upickle.default.reader[Seq[String]] ) items ==> Seq("HELLO", "WORLD") val rewritten = upickle.default.transform(items).to(CirceJson) val stringified = CirceJson.transform(rewritten, StringRenderer()).toString stringified ==> """["HELLO","WORLD"]"""

libraryDependencies += "com.lihaoyi" %% "ujson-play" % "0.9.5"

import ujson.play.PlayJson import play.api.libs.json._ val playJson: play.api.libs.json.JsValue = PlayJson( """["hello", "world"]""" ) val updatedPlayJson = JsArray( for(v <- playJson.as[JsArray].value) yield JsString(v.as[String].toUpperCase()) ) val items: Seq[String] = PlayJson.transform( updatedPlayJson, upickle.default.reader[Seq[String]] ) items ==> Seq("HELLO", "WORLD") val rewritten = upickle.default.transform(items).to(PlayJson) val stringified = PlayJson.transform(rewritten, StringRenderer()).toString stringified ==> """["HELLO","WORLD"]"""

Json4s

libraryDependencies += "com.lihaoyi" %% "ujson-json4s" % "0.9.5"

import org.json4s.JsonAST val json4sJson: JsonAST.JValue = Json4sJson( """["hello", "world"]""" ) val updatedJson4sJson = JsonAST.JArray( for(v <- json4sJson.children) yield JsonAST.JString(v.values.toString.toUpperCase()) ) val items: Seq[String] = Json4sJson.transform( updatedJson4sJson, upickle.default.reader[Seq[String]] ) items ==> Seq("HELLO", "WORLD") val rewritten = upickle.default.transform(items).to(Json4sJson) val stringified = Json4sJson.transform(rewritten, StringRenderer()).toString stringified ==> """["HELLO","WORLD"]"""

Cross-Library Conversions

uJson lets you convert between third-party ASTs efficiently and with minimal overhead: uJson converts one AST to the other directly and without any temporary compatibility data structures. The following example demonstrates how this is done: we parse a JSON string using Circe, perform some transformation, convert it to a Play-Json AST, perform more transformations, and finally serialize it back to a String and check that both transformations were applied:

import ujson.circe.CirceJson val circeJson: io.circe.Json = CirceJson( """["hello", "world"]""" ) val updatedCirceJson = circeJson.mapArray(_.map(x => x.mapString(_.toUpperCase))) import ujson.play.PlayJson import play.api.libs.json._ val playJson: play.api.libs.json.JsValue = CirceJson.transform( updatedCirceJson, PlayJson ) val updatedPlayJson = JsArray( for(v <- playJson.as[JsArray].value) yield JsString(v.as[String].reverse) ) val stringified = PlayJson.transform(updatedPlayJson, StringRenderer()).toString stringified ==> """["OLLEH","DLROW"]"""

uPack



uPack is uPickle's MessagePack library, which can be used to easily manipulate MessagePack source and data structures without converting them into Scala case-classes. This all lives in the upack package.

uPack comes bundled with uPickle, or can be used stand-alone via the following package coordinates:

libraryDependencies += "com.lihaoyi" %% "upack" % "0.9.5"

The following basic functions are provided in the upack package to let you read and write MessagePack structs:

def transform[T](t: Readable, v: upickle.core.Visitor[_, T]) = t.transform(v) /** * Read the given MessagePack input into a MessagePack struct */ def read(s: Readable): Msg = transform(s, Msg) def copy(t: Msg): Msg = transform(t, Msg) /** * Write the given MessagePack struct as a binary */ def write(t: Msg): Array[Byte] = { transform(t, new MsgPackWriter()).toByteArray } /** * Write the given MessagePack struct as a binary to the given OutputStream */ def writeTo(t: Msg, out: java.io.OutputStream): Unit = { transform(t, new MsgPackWriter(out)) } /** * Parse the given MessagePack input, failing if it is invalid */ def validate(s: Readable): Unit = transform(s, NoOpVisitor)

MessagePack structs are represented using the upack.Msg type. You can construct ad-hoc MessagePack structs using upack.Msg , and can similarly parse binary data into upack.Msg for ad-hoc querying and manipulation, without needing to bind it to Scala case classes or data types:

val msg = upack.Arr( upack.Obj(upack.Str("myFieldA") -> upack.Int32(1), upack.Str("myFieldB") -> upack.Str("g")), upack.Obj(upack.Str("myFieldA") -> upack.Int32(2), upack.Str("myFieldB") -> upack.Str("k")) ) val binary: Array[Byte] = upack.write(msg) val read = upack.read(binary) assert(msg == read)

You can read/write Scala values to upack.Msg s using readBinary / writeMsg :

val big = Big(1, true, "lol", 'Z', Thing(7, "")) val msg: upack.Msg = upickle.default.writeMsg(big) upickle.default.readBinary[Big](msg) ==> big

Or include upack.Msg s inside Seq s, case-classes and other data structures when you read/write them:

val msgSeq = Seq[upack.Msg]( upack.Str("hello world"), upack.Arr(upack.Int32(1), upack.Int32(2)) ) val binary: Array[Byte] = upickle.default.writeBinary(msgSeq) upickle.default.readBinary[Seq[upack.Msg]](binary) ==> msgSeq

You can also convert the uPack messages or binaries to ujson.Value s via upack.transform . This can be handy to help debug what's going on in your binary message data:

val msg = upack.Arr( upack.Obj(upack.Str("myFieldA") -> upack.Int32(1), upack.Str("myFieldB") -> upack.Str("g")), upack.Obj(upack.Str("myFieldA") -> upack.Int32(2), upack.Str("myFieldB") -> upack.Str("k")) ) val binary: Array[Byte] = upack.write(msg) // Can pretty-print starting from either the upack.Msg structs, // or the raw binary data upack.transform(msg, new ujson.StringRenderer()).toString ==> """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]""" upack.transform(binary, new ujson.StringRenderer()).toString ==> """[{"myFieldA":1,"myFieldB":"g"},{"myFieldA":2,"myFieldB":"k"}]""" // Some messagepack structs cannot be converted to valid JSON, e.g. // they may have maps with non-string keys. These can still be pretty-printed: val msg2 = upack.Obj(upack.Arr(upack.Int32(1), upack.Int32(2)) -> upack.Int32(1)) upack.transform(msg2, new ujson.StringRenderer()).toString ==> """{[1,2]:1}"""

Note that such a conversion between MessagePack structs and JSON data is lossy: some MessagePack constructs, such as binary data, cannot be exactly represented in JSON and have to be converted to strings. Thus you should not rely on being able to round-trip data between JSON <-> MessagePack and getting the same thing back, although round tripping data between Scala-data-types <-> JSON and Scala-data-types <-> MessagePack should always work.

Some of the differences between the ways things are serialized in MessagePack and JSON include:

Large Longs in JSON are represented as ujson.Str s if n > 2^53; in MessagePack, they are represented as upack.Int64 s or upack.UInt64 s

s if n > 2^53; in MessagePack, they are represented as s or s Array[Byte] s in JSON are represented as lists of numbers; in MessagePack, they are represented as upack.Binary

If you need to construct Scala case classes or other data types from your MessagePack binary data, you should directly use upickle.default.readBinary and upickle.default.writeBinary : these bypass the upack.Msg struct entirely for the optimal performance.

Performance



The uPickle has a small set of benchmarks in bench/ that tests reading and writing performance of a few common JSON libraries on a small, somewhat arbitrary workload. The numbers below show how many times each library could read/write a small data structure in 25 seconds (bigger numbers better). In some libraries, caching the serializers rather than re-generating them each read/write also improves performance: that effect can be seen in the (Cached) columns.

JVM Case Class Serialization Performance

uPickle runs 30-50% faster than Circe for reads/writes, and ~200% faster than play-json.

Library Reads Writes Reads (Cached) Write (Cached) Jackson-Scala 2,080,682 8,905,996 Play Json 1,123,923 1,518,832 1,360,201 1,763,854 Circe 2,172,638 2,057,883 2,519,163 2,416,923 upickle.default 3,078,442 4,018,176 3,564,215 4,749,887 upickle.default binary 4,907,232 6,812,322 6,332,698 9,367,948

As you can see, uPickle's JSON serialization is pretty consistently ~30% faster than Circe for reads and ~50% faster for writes. It is also faster than Jackson-Scala on reads by a similar margin, although somehow Jackson-Scala has write performance that far outstrips everyone else. uPickle JSON is 100-200% faster than Play-Json, depending on workload.

uPickle's binary MessagePack backend is then another 50-100% than uPickle JSON.

uPickle achieves this speed by avoiding the construction of an intermediate JSON AST: while most libraries parse from String -> AST -> CaseClass , uPickle parses input directly from String -> CaseClass . uPickle also provides a ujson.Js AST that you can use to manipulate arbitrary JSON, but ujson.Js plays no part in parsing things to case-classes and is purely for users who want to manipulate JSON.

JVM AST Serialization Performance

If you benchmark the various libraries parsing from String -> AST , instead of String -> CaseClass , uPickle's numbers are in line with everyone else's:

Library AST Reads AST Writes uJson 2,809,313 4,846,925 uPack (binary) 5,927,786 13,218,990 Play Json 3,372,847 3,291,835 Circe 5,744,314 5,669,830 Argonaut 2,189,890 2,699,175 Json4s Native 3,459,126 1,302,335

One thing to note is that uPickle's AST-serialization performance isn't any better than uPickle's CaseClass-serialization performance. This may seem surprising, but it makes sense when you realize that when serializing case classes, uPickle goes straight from the case class to the output string without converting to the AST in between

JS Case Class Serialization Performance

While all libraries are much slower on Scala.js/Node.js than on the JVM, uPickle runs 4-5x as fast as Circe or Play-Json for reads and writes.

Library Reads Writes Reads (Cached) Write (Cached) Play Json 117,808 201,701 148,964 268,816 Circe 136,181 461,873 160,860 678,861 upickle.default 666,626 970,840 931,945 1,462,505 upickle.default.web 581,024 970,332 931,505 1,734,548 upickle.default binary 601,333 497,954 931,505 1,734,548

Apart from the huge difference between uPickle and it's alternatives, this Scala.js/Node.js benchmark also shows a much larger effect for caching the picklers: 80-100% speedup. It seems likely that Scala.js/Node.js just isn't as good at optimizing away the overhead of temporary/throwaway data-structures, whether they're intermediate-ASTs or temporary-pickler-instances.

Another thing of note is the .web entries; these are benchmarking uPickle using the Node.js JSON.parse and JSON.stringify builtin functions, rather than our own parser. It turns out that these builtins appear to be slower than our parser for this benchmark, but they're available as the upickle.default.web.read/write functions or the upickle.WebJson object if you for some reason want to use them.

In Scala.js, uPickle's binary MessagePack backend isn't any faster than upickle JSON. This makes sense when you consider that Scala.js isn't optimized for handling binary data.

uJson is a fork of Erik Osheim's excellent [Jawn](https://github.com/non/jawn) JSON library, and inherits a lot of it's performance from Erik's work.

Version History



Bump Geny dependency

Add ability to parse from any geny.Readable data type, such as java.io.InputStream

Basic support for GADTs #288

ujson.Value and upack.Msg now support the geny.Writable interface

and now support the geny.Writable interface Added new upickle.default.writable and upickle.default.writableBinary , to serialize Scala data types via the geny.Writable interface

Improved performance by avoiding allocations in deserialization hot paths #284

Add null-safe cast helpers #274

Support for Scala 2.13.0 final

Introduced new MessagePack backend for binary serialization! This is used via upickle.default.writeBinary / upickle.default.readBinary , or via the standalone upack.read / upack.write package. Binary serialization typically is 50-100% faster than JSON when running on the JVM.

/ , or via the standalone / package. Binary serialization typically is 50-100% faster than JSON when running on the JVM. ujson.Js.Value , ujson.Js.Obj , etc. are now just ujson.Value , ujson.Obj

, , etc. are now just , Small Long 64-bit integers are now read/written as JSON numbers by default; only large values which cannot be precisely stored in Double -precision floating point (n > 2^53) are written as strings. You can revert to the old behavior via a Custom Configuration with: override implicit val LongWriter = new Writer[Long] { def write0[V](out: Visitor[_, V], v: Long) = out.visitInt64(v, -1) }

64-bit integers are now read/written as JSON numbers by default; only large values which cannot be precisely stored in -precision floating point (n > 2^53) are written as strings. You can revert to the old behavior via a Custom Configuration with: upickle.json.* and upickle.Js.* have been removed (use ujson.* .

Added the escapeUnicode: Boolean = false flag to ujson.Js#Render and ujson.Renderer ; pass in false to rrender unicode characters verbatim rather than escaping them.

Fix ability to construct single-element Js.Obj values without explicitly wrapping values [#230](https://github.com/lihaoyi/upickle/issues/230)

values without explicitly wrapping values [#230](https://github.com/lihaoyi/upickle/issues/230) Add JsValue#bool helper [#227](https://github.com/lihaoyi/upickle/pull/223) for extracting boolean values

Add ability to update a JSON dictionary key or array item in place via json(0)("myFieldA") = _.num + 100

Fix uJson direct dependency artifact naming

Added ujson.copy helper

helper Added implicit constructors for ujson.Js.Obj and Arr

Fix conversion of case classes to other case classes via upickle.default.transform

~3x faster than 0.5.1; uPickle now has the best Performance out of any of the commonly-used Scala JSON libraries

The old upickle.Js JSON AST and non/jawn dependency have been combined into the uJson standalone library. uJson provides high-performance streaming JSON processing that lets uPickle parse input strings directly to case classes without an intermediate AST.

JSON AST and non/jawn dependency have been combined into the uJson standalone library. uJson provides high-performance streaming JSON processing that lets uPickle parse input strings directly to case classes without an intermediate AST. upickle.Js objects are now mutable, and had some implicits added to make Construction less awkward.

objects are now mutable, and had some implicits added to make Construction less awkward. The set of Common Operations and JSON Utilities has been fleshed out: you now have convenient helpers for streaming JSON re-formatting or validation, and can read from arbitrary inputs (strings, byte-arrays, files, ...) using the same upickle.default.read call

call The way you write Custom Picklers and Custom Configuration has changed. The new ways are hopefully more intuitive, allow much better back-end performance, and should be just as flexible as the old way, but if you have custom picklers/configurations in your code you'll have to go update them.

uPickle now supports parsing to (and serializing) Other ASTs, from libraries such as Circe, Argonaut, Json4s or Play-Json. You can now also do high-performance/streaming Cross-Library Conversions to transform from one library's AST to another without any intermediate structures.

Strip out automatic "deep" case-class serialization: you must now define a serializer for each case class you want to deal with, preferably in the companion object

Upgrade Jawn version to 0.11.0, add helpers in ujson to parse JSON coming from files.

to parse JSON coming from files. New ujson.writeTo function for serializing JSON directly to a java.io.Writer , rather than creating a String

function for serializing JSON directly to a , rather than creating a ujson.write now takes an optional `sortKeys` flag, if you want the JSON dictionaries to rendered in a standardized order

now takes an optional `sortKeys` flag, if you want the JSON dictionaries to rendered in a standardized order Drop support fo Scala 2.10

Support for BigInt and BigDecimal , thanks to Jisoo Park

and , thanks to Jisoo Park Js.Value s are now serializable, thanks to Felix Dietze

s are now serializable, thanks to Felix Dietze Made it easy to write Manual Sealed Trait Picklers in order to work around problems with SI-7046

Changes to PPrint

Changes to PPrint

Allow custom handling for JSON null s via a Custom Configuration

s via a Custom Configuration Made upickle.key a case class

a Remove unnecessary dependency of derive on default arguments #143

on default arguments #143 Fixed derivation allowing Caching Picklers in a case class 's companion object, potentially speeding up compilation times and runtimes

Add .arr: Seq[Js.Value] , .obj: Map[String, Js.Value] , .str: String and .num: Double helper methods on Js.Value to simplify usage as a simple JSON tree.

, , and helper methods on to simplify usage as a simple JSON tree. Invalid.Json and Invalid.Data now have better exception messages by default, which should simplify debugging

and now have better exception messages by default, which should simplify debugging Some usages should compile faster due to fiddling with implicits (#138)

Tweaks to PPrint

You can now pass in an indent parameter to upickle.default.write in order to format/indent the JSON nicely across multiple lines

parameter to in order to format/indent the JSON nicely across multiple lines Derivation based on sealed abstract classes works, in addition to traits (#104), thanks to Voltir

Fix non-deterministic failure due to improperly implemented equals / hashCode in macro (#124), thanks to Voltir

/ in macro (#124), thanks to Voltir Slightly improve hygiene of uPickle/PPrint macro expansion

uPickle de-serialization failures should no longer throw MatchErrors (#101)

(#101) Using case-class-derived Reader s/ Writer s should no longer fail in class extends clauses (#108)

s/ s should no longer fail in class clauses (#108) Float.NaN and Double.NaN are now properly handled (#123)

and are now properly handled (#123) Provided an example of a Custom Configuration being used to snake_case case-class fields during serialization/de-serialization (#120)

Fix more bugs in PPrint derivation

Fix some bugs in PPrint derivation

Remove unnecessary shapeless dependency

Fix more edge cases to avoid diverging implicits

Fix more edge cases around typeclass derivation: #94, #95, #96

Don't get tripped up by custom Apply methods: #48

Fixed edge cases around typeclass derivation

Top-to-bottom rewrite of type-class derivation macros. Much faster, more reliable, etc.. Still one or two cases where it misbehaves, but much fewer than before. Extracted it into the derive subproject

subproject Force users to choose between import upickle.default._ which now renders sealed trait hierarchies as dictionaries with a $type attribute, and import upickle.legacy._ which does the old-style array-wrapper.

which now renders sealed trait hierarchies as dictionaries with a attribute, and which does the old-style array-wrapper. You can also now create your own custom subclass of upickle.Api if you wish to customize things further, e.g. changing the type-attribute or changing the rendering of case classes.

Support for java.util.UUID , which are serialized as strings in the standard format

Re-published for Scala.js 0.6.1

'Symbol s are now read/write-able by default

s are now read/write-able by default Added lots of warnings for common issues

Map[String, V] now pickles to a JSON dictionary "key": "value", ...} . Map[K, V] for all other K != String are unchanged

now pickles to a JSON dictionary . for all other are unchanged Source maps now point towards a reasonabel place on Github

Fixed [#23](https://github.com/lihaoyi/upickle/issues/23): self-recursive data structures are now supported.

Fixed [#18](https://github.com/lihaoyi/upickle/issues/18): you can now auto-pickle classes in objects that originated from traits that were mixed in.

Support reading and writing null

Fixed Reader/Writer macros for single-class sealed hierarchies

Used CanBuildFrom to serialize a broader range of collections

Added a pickler for Unit / ()

Swapped over from the hand-rolled parser to using Jawn / JSON.parse on the two platforms, resulting in a 10-15x speedup for JSON handling.

/ on the two platforms, resulting in a 10-15x speedup for JSON handling. Renamed Js.{String, Object, Array, Number} into Js.{Str, Obj, Arr, Num} , and made Js.Arr and Js.Obj use varargs, to allow for better direct-use.

into , and made and use varargs, to allow for better direct-use. Documented and exposed JSON API for direct use by users of the library.

Improved error messages for unpickle-able types

ScalaJS version now built against 0.5.3

Members of sealed trait/class hierarchies are now keyed with the fully-qualified name of their class, rather than an index, as it is less likely to change due to adding or removing classes

Members of sealed hierarchies and parameters now support a upickle.key("...") annotation, which allows you to override the default key used (which is the class/parameter name) with a custom one, allowing you to change the class/param name in your code while maintaining compatibility with serialized structures

annotation, which allows you to override the default key used (which is the class/parameter name) with a custom one, allowing you to change the class/param name in your code while maintaining compatibility with serialized structures Default parameters are now supported: they are used to substitute missing keys when reading, and cause the key/value pair to be omitted if the serialized value matches the default when writing

Missing keys when deserializing case classes now throws a proper Invalid.Data exception

exception object s are now serialized as {} rather than [] , better matching the style of case classes

s are now serialized as rather than , better matching the style of case classes 0-argument case classes, previously unsupported, now serialize to {} the same way as object s

the same way as s Fixed a bug that was preventing multi-level sealed class hierarchies from being serialized due to a compilation error

Fixed a bug causing case classes nested in other packages/objects and referred to by their qualified paths to fail pickling

Tightened up error handling semantics, swapping out several MatchError s with Invalid.Data errors

Cleaned up the external API, marking lots of things which should have been private private or stuffing them in the Internals namespace

namespace Organized things such that only a single import import upickle._ is necessary to use the library

Tuples and case classes now have implicit picklers up to an arity limit of 22.

Case classes now serialize as JSON dictionaries rather than as lists.

Simple case classes and case class hierarchies are now auto-serializable view Macros. No need to define your own implicit using Case0ReadWriter anymore!

Serialize numbers as JSON numbers instead of Strings.