Setup & Deployment

Oracle Event Hub Cloud

We wont go through the entire details since this is straightforward and well documented. Here are the high level steps

create a Kafka broker(single node , colocated with Zookeeper) — documentation

create the topic (to store the tweets) — documentation

You’ll end up with something like this

Managed Kakfa broker

Kafka topic

Creating custom access rule

You would need to create a custom Access Rule to open port 6667 on the Kafka Server VM on Oracle Event Hub Cloud. This is to allow Kafka Connect Twitter (source) connector to work (details to follow)

Our stream processing app on Oracle Application Container handles the connectivity internally and does not need any ports to be opened

Navigate to Access Rules section

Create the new rule

New Access Rule created successfully

Oracle Application Container Cloud

The only thing you need to do in this case is to provision the Oracle Application Container Cloud Caching service — here is the documentation

Oracle ACCS Cache (serves as stream processing state store)

Oracle Developer Cloud

You can refer to previous blogs for the same (some of the details specific to this example will be highlighted here)

References

Deployment Configuration

Deployment config part 1

Deployment config part 2

Deployment confirmation in Oracle Developer Cloud

Deployments in Oracle Developer Cloud

Post-deployment status in Oracle Application Container Cloud

Oracle Application Container Cloud (applications list)

Environment variables for our application

Kafka Connect Twitter connector

You can read the details on Github project — here is a quick summary of how to setup the connector for consuming tweets from Twitter and relaying them to a Kafka topic of our choice

create a Twitter application

install Kafka on your local machine (this is required since Kafka Connect comes bundled with the Kafka distribution)— just download and unzip it !

clone the project — git clone https://github.com/Eneco/kafka-connect-twitter.git

build it — mvn clean package

configure connect-simple-source-standalone.properties — you can stick the defaults except for the Oracle Event Hub Kafka cluster info (highlighted)

bootstrap.servers=<oracle_ehcs_IP>:6667

key.converter=org.apache.kafka.connect.storage.StringConverter

value.converter=org.apache.kafka.connect.storage.StringConverter

key.converter.schemas.enable=true

value.converter.schemas.enable=true

internal.key.converter=org.apache.kafka.connect.json.JsonConverter

internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false

internal.value.converter.schemas.enable=false

offset.storage.file.filename=/tmp/connect.offsets

offset.flush.interval.ms=10000

configure twitter-source.properties — the highlighted attributes are the ones which need to populated as per your environment