In TiKV, we use gRPC for network communication, and we also have developed a Rust gRPC library — grpc-rs, which wraps the Google C gRPC with Rust Futures, and provides an ergonomic API for outside use.

But there are some problems for grpc-rs:

grpc-rs is not pure Rust

Its performance is not very good. It is slower than our customized protocol.

Sadly, we even met some panic in gRPC in production environments before.

So sometimes, we want to embrace the Rust community, and use a gRPC library fully written in Rust.

Of course, we don’t want to reinvent the wheels. Luckily, there are two existing libraries — grpc-rust and tower-grpc. Before we developed grpc-rs, we tried grpc-rust for some time, but this library was not stable before so we gave up and never use it again. Maybe it has already become better :-)

So now we pay attention to tower-grpc. Although tower-grpc is new and not stable too, it still has some benefits:

The authors of tower-grpc are famous in the Rust world, like carllerche who develops mio , and seanmonstar who develops hyper .

, and seanmonstar who develops . It is based on tokio, and we think tokio is the future of the network programming in Rust.

For us, the earlier embracing tower-grpc, the closer we cooperate with Rust community, and of course, we can improve tower-grpc and tokio at the same time.

But now, there is a problem for TiKV to try tower-grpc. TiKV uses rust-protobuf to encode/decode protobuf, but tower-grpc uses prost. Changing all associated codes is not an easy work, and needs lots of time. But we can still try tower-grpc simply.

At first, we need to clone kvproto — the definition of TiKV gRPC protocol, and create a tower directory:

git clone https://github.com/pingcap/kvproto.git

cd kvproto

mkdir tower

Refer to tower-grpc example Cargo.toml, we can create a similar Cargo.toml.

Notice: I met build error when built tower-grpc with Rust version rustc 1.29.0-nightly (4f3c7a472 2018-07-17) , if you meet the same problem, you need to clone the tower-grpc, and add following lines to tower-grpc/src/lib.rs , then use the local tower-grpc in your example.

#![feature(extern_prelude)]

#![feature(crate_in_paths)]

Then we create build.rs :

extern crate tower_grpc_build; fn main() {

tower_grpc_build::Config::new()

.enable_server(true)

.enable_client(true)

.build(&["../proto/tikvpb.proto"], &["../proto", "../include"])

.unwrap_or_else(|e| panic!("protobuf compilation failed: {}", e));

// Same for pdpb.proto and other proto files

}

At last, we create an empty file src/lib.rs and run cargo build , which generates the Rust files of the gRPC protocol in the out directory like target/debug/build/tower-9000a2ba77585c7e/out , we need to copy them to other directory manually for later use (of course, we can do this automatically in build.rs later).

The generated file, e.g. metapb.rs looks like below:

#[derive(Clone, PartialEq, Message)]

pub struct Cluster {

#[prost(uint64, tag="1")]

pub id: u64,

/// max peer count for a region.

/// pd will do the auto-balance if region peer count mismatches.

///

/// more attributes......

#[prost(uint32, tag="2")]

pub max_peer_count: u32,

}

You can’t import this file and use it directly, you need to include it, like:

pub mod metapb {

include!("proto/metapb.rs");

}

Now, we can create another project, refer to client.rs, we create a TiKV client, which will send raw_put command to TiKV:

pub fn main() {

let _ = ::env_logger::init(); let uri: http::Uri = format!("http://127.0.0.1:20161").parse().unwrap(); let h2_settings = Default::default();

let mut make_client = client::Connect::new(Dst, h2_settings, DefaultExecutor::current()); let say_hello = make_client.make_service(())

.map(move |conn| {

use tikvpb::client::Tikv;

use tower_http::add_origin; let conn = add_origin::Builder::new()

.uri(uri)

.build(conn)

.unwrap(); Tikv::new(conn)

})

.and_then(|mut client| {

use kvrpcpb::{Context, RawPutRequest};

use metapb::{RegionEpoch, Peer}; let ctx = Context{

region_id: 2,

region_epoch: Some(RegionEpoch{

conf_ver: 1,

version: 1,

}),

peer: Some(Peer{

id: 3,

store_id: 1,

is_learner: false,

}),

term: 6,

priority: 0,

isolation_level: 0,

not_fill_cache: false,

sync_log: false,

handle_time: false,

scan_detail: false,

}; client.raw_put(Request::new(RawPutRequest {

context: Some(ctx),

key: b"a".to_vec(),

value: b"123".to_vec(),

cf : "default".to_string(),

})).map_err(|e| panic!("gRPC request failed; err={:?}", e))

})

.and_then(|response| {

println!("RESPONSE = {:?}", response);

Ok(())

})

.map_err(|e| {

println!("ERR = {:?}", e);

}); tokio::run(say_hello);

}

We forcibly send command to Region 2, you can get the region information through tools pd-ctl and tikv-ctl.

After we run the client, the output is:

RESPONSE = Response { http: Response { status: 200, version: HTTP/2.0, headers: {"content-type": "application/grpc", "grpc-accept-encoding": "identity,deflate,gzip", "accept-encoding": "identity,gzip"}, body: RawPutResponse { region_error: None, error: "" } } }

We use tikv-ctl to query the data:

tikv-ctl --host 127.0.0.1:20161 print -k za

value: 123

Every key written to TiKV will be added z as the prefix, so here we use za , and we can see the value is “123”. Great, we have inserted a key-value successfully into TiKV.

As you can see, using tower-grpc is easy, and we have been considering migrating to tower-grpc https://github.com/tikv/tikv/issues/3951. If you are interested in this, feel free to contact us.