Idarlington

Not so common Kafka CLI commands

Tue, 30 Oct 2018 00:00:00 +0000

Recently, a colleague asked how to view keys of consumed records and I wasn’t sure it was possible. I thought I knew all the important cli commands then. Later on, after a google search, I found it was possible.

I decided to spend some time going through the Kafka console classes I use the most — kafka.tools.ConsoleConsumer and kafka.tools.ConsoleProducer.

Here are some not so common commands you can use in the CLI:

Kafka Console Consumer

print.key:

This property configures the console consumer to print the keys of the messages it consumes. The default value is false.
print.value:

When you are not concerned with printing the values of the records being consumed, you can set this property which configures the console consumer to/not to print values of records being consumed. The default value is true.
print.timestamp:

You can configure the console consumer to/not to print the timestamp of records being consumed. The default value is false. NO_TIMESTAMP is printed out if there is no timestamp configured for the records and the property is set to true.
key.separator:

This property configures the character that separates the key and value when they are both printed. The default value is the tab character \t.
line.separator:

Property configuring the character separating each record being consumed. The default value is \n.
formatter:

The name of the class to use for formatting kafka records being printed. The default value is kafka.tools.DefaultMessageFormatter. Other formatters are: kafka.tools.LoggingMessageFormatter, kafka.tools.NoOpMessageFormatter, kafka.tools.ChecksumMessageFormatter.

Example of console consumer command using the flags and properties defined above:

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
    --topic test-topic \
    --from-beginning \
    --formatter kafka.tools.DefaultMessageFormatter \
    --property print.key=true \
    --property print.value=true \
    --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
    --property value.deserializer=org.apache.kafka.common.serialization.StringSerializer

Kafka Console Producer

key.separator:

By default inputs to the Kafka producer console are recognised as values and null is used for the keys. This property allows users input key and value messages in the producer console. It represents the character separating the key and value. The default value is the \t character. It has to be used with the parse.key property.
parse.key:

This property indicates if the key is to be serialized, it is used with the key.separator property. The default value is false.
ignore.error:

This property configures the console producer to ignore errors when the key.separator property is set and there is not used in an input line in the console.

Example of console producer command using the flags and properties defined above:

kafka-console-producer.sh \
  --broker-list localhost:9092 \
  --topic test-topic \
  --property "parse.key=true" \
  --property "key.separator=-"


key1-value1
key2-value2
key3-value3

Error handling in Scala

Sat, 01 Jul 2017 00:00:00 +0000

Slides for the presentation I gave at Lagos Scala July Meetup.

When you dont feel like coding but have to

Mon, 22 May 2017 00:00:00 +0000

Slides for the presentation I gave at Lagos Agile May Meetup.

Getting started with Apache Avro

Thu, 30 Mar 2017 00:00:00 +0000

What is Avro ?

Avro is a data serialisation and RPC system like protobuff and thrift. It relies on a schema-based system. This schema is in JSON which is an advantage as most languages already have JSON libraries. Avro was originally developed by Doug Cutting to provide data serialisation and data exchange services for hadoop. It has since evolved to serve other technologies apart from hadoop.

Theoretically, Avro can be used in any language but it has APIs for PHP, Java, Perl, Python, C, C++, C#, Go, Haskell and Ruby.

Where Avro shines

Comparing Avro with similar systems (protobuff, thrift, message pack), these are the areas where it differs :

Dynamic typing:

Serialisation and deserialization can be done without code generation. Code generation is also available for statically typed languages as an optional optimisation.
Schema evolution:

Avro requires schemas when data is written or read. Most interesting is that you can use different schemas for serialisation and deserialization, and Avro will handle the missing/extra/modified fields.
Untagged data:

Other serialisation systems like protobuff tag data but providing a schema with binary data allows each datum be written without the overhead. The result is more compact data encoding and faster data processing.

Why you need Avro (or serialisation libraries generally):

You save storage space. Uber drastically reduced their storage needs by implementing data serialization and compression.
It significantly reduces the amount of time spent processing data.
Translates to less money saved on provisioning hardware.

Data types in Avro

Avro supports a range of data types. These types could be either primitive, complex or logical types.

Primitive types:

null : no value
boolean : a binary value
int : 32-bit signed integer
long : 64-bit signed integer
float : single precision (32-bit) IEEE 754 floating-point number
double : double precision (64-bit) IEEE 754 floating-point number
bytes : sequence of 8-bit unsigned bytes
string : Unicode character sequence

Complex Types:

Avro also provides complex data types which are majorly a combination of primitive types. The complex types are:

Records:

A record is a collection of multiple types. A record is equivalent to a JSON object or a dictionary in python.

{
    "type":"record",
    "name":"Point",
    "fields":[
        {
            "name":"x",
            "type":"int"
        },
        {
            "name":"y",
            "type":"int"
        }
    ]
}

Enum:

An enumeration is a list of items in a collection.

{
    "type":"enum",
    "name":"Suit",
    "symbols":[
        "SPADES",
        "HEARTS",
        "DIAMONDS",
        "CLUBS"
    ]
}

Arrays:

{ " type " : " array ", " items " : " int " }

Maps:

Map keys are assumed to be strings. Eg :
```
{"type": "map", "values": "long"}
```
Unions:

Unions are represented using JSON arrays.

["null", "string"] declares a schema which may be either a null or string.
Fixed:

This data type is used to declare a fixed-sized field that can be used for storing binary data. It has field name and data as attributes. Name holds the name of the field, and size holds the size of the field.

Defining a schema

As indicated, Avro schemas are defined with JSON. The Schema is a representation of the data to be serialized in avro data types.

Creating a schema for this Object

{
    "products":[
        {
            "id":1,
            "name":"A green door",
            "price":12.50,
            "tags":[
                "home",
                "green"
            ]
        },
        {
            "id":1,
            "name":"A green door",
            "price":12.50,
            "tags":[
                "home",
                "green"
            ]
        },
        {
            "id":1,
            "name":"A green door",
            "price":12.50,
            "tags":[
                "home",
                "green"
            ]
        }
    ]
}

would result in :

{
    "namespace":"com.idarlington.avro",
    "type":"record",
    "name":"stream",
    "fields":[
        {
            "name":"products",
            "type":{
                "type":"array",
                "items":{
                    "name":"product",
                    "type":"record",
                    "fields":[
                        {
                            "name":"id",
                            "type":"integer"
                        },
                        {
                            "name":"name",
                            "type":"string"
                        },
                        {
                            "name":"price",
                            "type":"float"
                        },
                        {
                            "name":"tags",
                            "type":{
                                "type":"array",
                                "items":"string"
                            }
                        }
                    ]
                }
            }
        }
    ]
}

To validate and test your avro schema, you can check this out.

My next post would be on serialzing and deserializing with Avro. If you like this post, please share ♥.

MongoDB Replication

Mon, 14 Nov 2016 00:00:00 +0000

Replication?

Replication provides redundancy and increases data availability. With multiple copies of data on different database servers, replication provides a level of fault tolerance against the loss of a single database server.

— MongoDB Manual

A replica set in MongoDB is a group of mongod instances that maintain the same data set. The mongod instances are known as nodes. Nodes could be primary or secondary.

By default write operations are performed on the primary node and synced to other secondary nodes. Read operations are also performed on the primary node by default but the configuration on read operations can be changed.

Benefits of replication

High availability & redundancy
Distributes database requests, therefore, could increase application’s speed (has to be configured)

Setting Up a Replica Set

Set Up MongoDB on the servers :

Mongod instances need to be setup on the servers you plan to use in the replica set.

Ideally, a replica set is to have an odd number of mongod instances; this is to ensure that during an election for a master mongod instance, quorom can be reached. If you can only set up even mongod instances, then you have to add an arbiter. The minimum number of mongod instances is three, if you can only setup two, then you have to add an arbiter.

Arbiters are mongod instances that are part of a replica set but do not hold data. Arbiters participate in elections in order to break ties.

I have seen a few cases where all mongod instances of a replica set are setup on a single server, I think it beats the purpose of setting up a replica set. If that server goes down, then your MongoDB server becomes unavailable.
Check connection between Mongod instances :

The connection between the Mongod instances on the different servers needs to be tested and confirmed. This is important if you use a firewall on your servers. I have had cases where I setup a replica set and it wasn’t working only to find out the connection port between the Mongod instances was blocked.

The hostname of the servers where other Mongod instances are located needs to be added to each server’s /etc/hosts. For a server with hostname idarlington and ip 192.168.2.218, it should be added as :
```
192.168.2.218 idarlington
```
Set up configuration file :

After confirming the connection between the Mongod instances, you can now begin the replica set setup proper. You need to edit the MongoDB conf file that starts the Mongod service.

For most linux systems the conf file is located in the /etc/init. For MongoDB 2.4 and lower, the file is mongodb.conf while for MongoDB 2.6 and above, the file is mongod.conf

For MongoDB version 2.6 and above
```
replication.replSetName : "replset name you have chosen"
```
OR
```
replication:
   replSetName : "replset name you have chosen"
```
For MongoDB 2.4 and below
```
replSet  = "replset name you have chosen"
```
Initiate Replica Set :

You can now exit the conf file and start a mongo shell on the server you want to be the primary node.
```
  Mongo
  
```
Then to initiate the replica set :
```
rs.initiate()
```
To show the configuration of the replica set :
```
 rs.conf() 
```
To show the recent status of the replica set :
```
rs.status()
```
Add other members :

After initiating the replica set, its only member is the current server where you are running the mongo shell. To add other members to the replica set, use the IP address or hostname of the server with the mongod instance you want to add like
```
  rs.add("192.168.2.128")
  
```
Adding an arbiter :

Arbiters are useful when you have an even number of nodes, if you already have an odd number of nodes, it is not needful. To add an arbiter, you can either use its hostname or ip address.
```
  rs.addArb("192.168.2.13")
  
```

Notes

If you formerly built your application to work with a standalone MongoDB server, you have to update your mongodb connection to work with a replica set. The way to do this differs with the ORM or driver you use for your application.
You can set read preference on your application. The available options are primary, primaryPrefered, secondary, secondaryPrefered, nearest.

Why Scala ?

Mon, 24 Oct 2016 00:00:00 +0000

Slides for the presentation I gave at Lagos Scala October Meetup.

Best resources for learning Scala

Thu, 14 Jul 2016 00:00:00 +0000

Assuming I have been able to convince you to learn Scala, as a good friend, I won’t throw you into the deep end with no guide.

Here are few resources I believe will help you fully learn Scala and accelerate the process :

Atomic Scala : This book assumes the reader has no prior programming experience. It introduces solution patterns and syntax. It provides the reader with the foundations of programming and Scala. It could be the first book you use in learning Scala but not the last.
Big Data University’s Introduction to Scala : This is a series of videos on Scala that can be completed within 6-8 hours. It is assumed you have no prior knowledge of Scala but have some programming experience. The videos were created by Typesafe (now known as Lightbend)
Learning Scala : This book is a comprehensive introduction for beginner and intermediate programmers to Scala. It contains syntax diagrams, examples, and exercises. You’ll start with Scala’s core types and syntax before diving into higher-order functions and immutable data structures. This book is a great place to start for student, intermediate or experts that aren’t familiar with the Java stack and is about 220 pages long.
Scala Cookbok : Although I wouldn’t advise anyone to use a cookbook in learning a language or framework as they give the learner a shallow understanding of the available features and design decisions.

This book gives the reader an overview of the language, it can also be used as a quick reference which is what I use it for most times. It is well written and approachable.
Joel Abrahamsson’s Learning Scala : This is a series of blog posts on Joel Abrahamsson’s adventures in learning scala. It is a short series but gives the reader a easy way to approach learning Scala. It is mostly suited for novices and beginners.
Functional Programming Principles in Scala on Coursera : This coursera course is taught by Martin Odersky - the creator of scala. It focuses on programming in Scala using the functional paradigm and gives a great overview of the language. The course could span about seven weeks and is suited for intermediate programmers.
Programming Scala Programming Scala gets you up to speed on Scala, it is also packed with code examples. This book shows you how to be productive with the language and ecosystem and is quite comprehensive and easy to read.
Programming in Scala : This book is written by some of the core maintainers and developers of Scala - Martin Odersky, Lex Spoon, and Bill Venners . It is a detailed and definitive guide to Scala. It provides great insights on the language’s features with detailed examples. There is a free online version of the book.
Twitter Scala School : This resource started as a series the of lectures at twitter to prepare experienced engineers to be productive Scala programmers. The lessons are quite concise and properly organized. It is suited for intermediate programmers who have programming experience and quickly want to grasp Scala concepts.
Scala Tour : This is similar to twitter Scala School in its concise nature. It provides an overview to the main features of Scala. It also allows you to run and test your code as you learn.
Scala Exercises : I believe this resource should be used by anyone learning Scala. It is a set of Scala exercises that have sections on Scala features. It is an open source project actively maintained by 47 Degrees. It is one of my best Scala resources and one that I have contributed to.

As you progress with learning scala, you can try solving the 99 problems and project euler problems using scala.

Another good resource is pavel fatin’s Scala Collections Tips and Tricks.

Know of other awesome resources ? have questions ? please do comment and share.

Wishing you a smooth Journey in Scala ; )

Why you should learn Scala

Sun, 10 Jul 2016 00:00:00 +0000

I want to believe you must have heard or read about Scala and learnt it was a JVM language. You probably thought - just another JVM language that would die out soon. Right ? Or maybe you haven’t heard of Scala. How so?

History

Scala is short for scalable language, it combines both object oriented and functional programming concepts. It runs on the JVM (Java Virtual Machine). Scala was created by Martin Odersky (he worked on the Java compiler) and it’s first release was in 2003.

In 2011, corporate stewardship was brought to bear – a set of people were making sure that the language was enterprise-ready at all times. Scala is used by some of the largest companies and startups LinkedIn, Twitter, Foursquare, The Guardian & Intel to name a few.

I have been using Scala for over a year and I believe it won’t die out. I would like to give you reasons to ~~ditch that other language and~~ start using Scala. I would be presenting my case using powerful features in Scala known as case classes (no pun intended) and pattern matching. Pattern matching is described as switch on steroids on Scala’s official website. I think comparing it with switch cases is quite an injustice to it. It has also been called Swiss army knife of Scala.

case Concise Codes

Unlike Java that is riddled with boiler plate codes, Scala is quite concise. Though to be fair, using Java 8 lambdas could make Java code a bit concise.

Comparing codes creating a user class in Scala and Java clearly shows Scala’s concise nature.

//Java Code
public class User {
  private String surname;
  private String email;
  public User(String name, String surname, String email) {
    this.name = name;
    this.surname = surname;
    this.email = email;
  }
  public void setName(String name) {
    this.name = name;
  }
  public void setSurname(String surname) {
    this.surname = surname;
  }
  public void setEmail(String email) {
    this.email = email
  }
  public String getName() {
    return this.name;
  }
  public String getSurname() {
    return this.surname;
  }
  public String getEmail() {
    return this.surname;
  }
}

//Scala Code

class User(
 var name: String,
 var surname: String,
 var email: String)

When we started Spark, we wanted it to have a concise API for users, which Scala did well.

— Matei Zaharia,CTO @ Databricks

case Functional & OOP

Scala mixes the functional paradigm with pure object orientation. In Scala, functional programming is the recommended way to program but the creators knew making that deep dive would be tough for anyone so you can still use your OOP patterns.

Functions are first class objects in Scala. Using functional programming, you will be able to write less code that is reusable and easily maintained.

Solving the first Euler problem (Find the sum of all the multiples of 3 or 5 below 1000) in Scala with the functional paradigm shows clean and maintainable code.

val result = (1 until 1000).view.filter(n => n % 3 == 0 || n % 5 == 0).sum

case Static Typing and Type Inference

Statically typed programming languages do type checking (the process of verifying and enforcing the constraints of types) at compile-time as opposed to run-time.

Scala’s static typing is one of my favorite feature.It has been proven that static typing improves code quality. This is important for data scientists, if you have a large job that will run for hours, you don’t want to find out half way through the job that it was not a correct implementation.

Scala’s static typing helps you write less tests, you don’t need to write tests to check type consistency as the compiler does that for you (Sigh of relief). This helps you reduce the amount of stupid bugs at an early stage.

Also, with Scala’s type inference you can create variables without a specified type and the type would be inferred.

case Java Interoperability

Scala has awesome Java interoperability which means you can import and use your Java codes and libraries in your Scala application (isn’t that awesome?). The good thing is that integration is seamless as Scala runs on the JVM, you do it without any performance penalty.

case Concurrency and Distribution

Scala is suited for multi-core programming, it has built in tools for its implementation. Using Immutable values and collections is the recommended method of programming in Scala, as it guards your data from being mangled by different processes.

Scala also uses Akka which is a toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications. Scala’s functional nature makes it easier to write safe and performant multi-threaded code. There is less reliance on mutable state and Scala’s futures and actors provide powerful tools for organizing concurrent systems.

case High Paying Jobs and Most loved Language

From the results of the recent stack-overflow survey, Scala is amongst the top five most loved languages. Scala has recently seen adoption as the language for introduction to computer science in some educational institutions.

It is no mistake that Spark (a data engine written in Scala) jobs and Scala jobs are the top paying tech jobs in the US and amongst the top ten paying tech jobs worldwide according to the stack-overflow survey. I guess if none of the above cases impressed you, this should do right?

case _

Other features that make Scala awesome are

Options – No more NullPointerExceptions code crashes or errors.
One class != one file as you can have more than a class in a file.
Scala traits - multiple inheritance without some of it’s dangers but with some limitations.
Scala streams - an awesome scala collection that can be used for efficient Big Data pocessing.
Case classes and pattern matching.
Semicolons are not compulsory.

So, what do you think? Drop your thoughts in the comments section.