Go to the profile of Rahul Muttineni
Rahul Muttineni
CTO at TypeLead, Inc. (Techstars '17)

Integrating Eta Into Your Scala Projects

via an sbt plugin

Connecting Eta and Scala using sbt

We’ve already shown you how to pull IN Haskell packages, and how to pull IN Java code from Eta. In this post, we’ll show you how to conveniently push OUT Eta code to Scala using an sbt plugin.

Prerequisites

Motivation

To provide a context around why you’d want to integrate Eta into your Scala projects in the first place, we’re going to setup a project around a specific problem. This project is available on GitHub if you want to see it directly, but we’ll build it up from scratch so that you understand how it pieces together.

Imagine we have an existing Scala web service that tracks some data and writes it to a database. But we find that it’s slowing down our response time, so we scale out by building a new microservice that handles the data instead and takes care of the database transaction. Moreover, we use a communication channel (say Kafka) to communicate the data for processing.

In this post, we’ll be simply reading from and writing to a file to make it easy to run locally, but with minimal effort, we can modify the project and make it a microservice that’s a Kafka Consumer.

Now let’s talk about how we’re going to process the data.

Here’s some sample data:

{
    "id": 1,
    "first_name": "Jerrold",
    "last_name": "Hache",
    "email": "jhache0@ca.gov",
    "gender": "Male",
    "ip_address": "149.180.165.206",
    "events": {
        "clicks": 99
    }
}

NOTE This has been pretty printed for your reading pleasure, but each individual JSON object will be compacted into a single line.

As you can see, we have some user data, with the primary payload being the number of times they’ve clicked on something interesting.

Let’s assume there was a bug in the old version of the main service which gave us an inaccurate number of clicks — in particular, 50 less than the actual number.

The job of this microservice will be to:

  • Make the first_name field all capital letters
  • Make the last_name field all capital letters
  • Fix the number of clicks by adding 50
  • Send it out for further processing

Project Setup

  1. Start a new SBT project.
> sbt new sbt/scala-seed.g8
....
Minimum Scala build.

name [My Something Project]: example

Template applied in ./example

The project structure should look something like this:

example
- src
  - main
    - scala
      - example
        - Hello.scala
  - test
    - scala
      - example
        - HelloSpec.scala
- project
 - build.properties
 - Dependencies.scala

2. Add the sbt-eta plugin.

  • Create a new fileproject/plugins.sbt
  • Add the following line:
addSbtPlugin("com.typelead" % "sbt-eta" % "0.1.0")

3. Setup an Etlas project.

  • Create the src/main/eta directory tree and make that the current working directory.
> mkdir -p src/main/eta
> cd src/main/eta
  • Initialize an Etlas project by following these instructions and the differences shown below:
> etlas init
Package name? [default: eta] example
...
What does the package build:
   1) Library
   2) Executable
Your choice? 1
...
Source directory:
 * 1) (none)
   2) src
   3) Other (specify)
Your choice? [default: (none)] 1
...
Guessing dependencies...
Generating LICENSE...
Generating Setup.hs...
Generating ChangeLog.md...
Generating example.cabal...

4. Copy the contents of this data file to src/main/resources/data.json.

Great! Now your project structure is ready. It should look something like this:

example
- src
  - main
    - eta
      - ChangeLog.md
      - LICENSE
      - Setup.hs
      - example.cabal
    - scala
      - example
        - Hello.scala
  - ...
- project
 - plugins.sbt
 - ...

Let’s take a look at how to solve the problem from above.

Implementation

  1. Create src/main/eta/Example/Transform.hs with the following contents:

If you haven’t seen it already, there’s a simple example at the end of the Haskell integration post that explains similar Eta code line-by-line. We’ll only be explaining the bits that are different from that example, so make sure you read it first.

Line 3

import Java

The Java module is in the standard library and it provides many helper functions for working with the Java Foreign Function Interface (FFI).

Line 11

fixJson :: JString -> JString

This will be the main function that does all the work. The type signature indicates that it takes a JString and returns a JString. A JString is an Eta type that maps to Java’s java.lang.String . It is what is referred to as a JWT or Java Wrapper Type. These types are used when working with the Java FFI. Consult the documentation or the blog post series on the FFI for more information.

Line 12

fixJson = toJava . transform . fromJava

This is the implementation of the fixJson function. Note that it makes no mention of its argument! This is because fixJson expresses what it does through a pipeline of functions. These pipelines, when constructed with the . operator (function composition), are read from right-to-left.

It means:

  • Take the input JString of fixJson and pass it to fromJava
  • Take the result of fromJava and pass it to transform
  • Take the result of transform and pass it to toJava
  • Take the result of toJavaand make it the output of fixJson

Note how concise this code is. This is because Eta is a functional-first language and the basic patterns of functional programming are extremely easy to express and clear to read.

Line 13

  where transform :: Text -> Text

Here, we use the where keyword to declare helper functions or expressions. This allows us to break apart the large fixJson function into invidual pieces and glue them together using the . operator at the end.

Text is Eta’s equivalent of Java’s java.lang.String and fromJava and toJava are overloaded functions that allow you to convert between JWTs and Eta types.

Line 14–16

transform json = 
  json & (key "first_name" . _String) %~ T.toUpper 
       & (key "last_name"  . _String) %~ T.toUpper                        
       & (key "events" . key "clicks" . _Integral) %~ (+ 50)

As explained at the end of the previous post, this uses the lens DSL from Haskell to declaratively express the transformations rules.

Some interesting features of this code:

  • All the expressions are strongly typed. If you had tried to glue together lenses that don’t make sense, the Eta compiler will not allow your program to compile.
  • No need to check whether a field exists in the JSON, check if a field is null, or check if it has the right type. The underlying implementation will simply not apply the changes if it doesn’t find the match. Which means if you have data with a missing first_name field, this code won’t throw an exception — instead it will simply not apply the uppercase transformation on the string and move on.

Line 18–19

foreign export java "@static eta.example.Transform.fixJson"
  fixJson :: JString -> JString

This is one of the most powerful features of Eta. Not only can you import Java methods into Eta, but you can also export them so that you can use them from any other JVM language.

This allows you to embed Eta into any of your existing projects that use a JVM language.

This declaration in particular will generate a new Java class of this form:

package eta.example;
public class Transform {
  public static String fixJson(String arg) {
    // Code generated by Eta compiler to invoke the Eta 
    // runtime system
  }
}

For more information on foreign export declarations, consult the documentation.

2. Update src/main/eta/example.cabal to add the dependencies to the libraries needed to compile the program from (1) and also mention a couple of other configuration options.

...
library
  exposed-modules:     Example.Transform
  build-depends:       base >=4.8 && <4.9
                     , aeson
                     , lens-aeson
                     , lens
                     , text
  default-language:    Haskell2010
  default-extensions:  OverloadedStrings

3. Rename src/main/scala/example/Hello.scala to src/main/scala/example/Main.scala and replace the contents:

package example
import java.io.File
import java.io.PrintWriter
import scala.io.Source
import eta.example.Transform
object Main extends App {
  val writer = new PrintWriter(new File("output.json"))
  Source.fromResource("data.json").getLines.foreach { json =>
    writer.write(Transform.fixJson(json) ++ "\n")
  }
  writer.close()
}

This will read from a the data.json resource and process it line-by-line, calling the fixJson method that we exported in the previous steps and writes the resulting output to the output.json file which will be created in the directory in which this program is run.

4. Open the terminal at the root of your sbt project and run sbt:

> sbt
...
[info] [etlas] Checking Maven dependencies...
...
[info] [etlas] Resolving dependencies...
...

If this is your first time using Eta, note that it will take some time to download and compile all your Eta dependencies, so wait patiently. Future invocations will be faster.

Once that’s done, you’ll be inside the sbt console, and you can compile and run the program.

> compile
...
> run
...

Check for the resulting output.json and you should see the JSON successfully transformed!

Your Turn

Now that you’ve seen how to integrate Eta and Scala, think of ways to introduce it into your projects.

Here are some questions you can ask yourself if you want to guage whether Eta will help you:

  • Do you have projects that deal with lots of complex data transformation?
  • Do you have projects that can benefit from static type checking and converting runtime errors into compile-time errors without getting in your way?
  • Do you want to spend less time testing and debugging and more time writing code with substance?
  • Do you want to push more work to the compiler so that you can sip coffee?

If you answered “Yes” to any of those questions, you’ll find that Eta will be extremely helpful.

If you have any issues, please reach out to us!

Gitter / IRC / Slack / Google Group / GitHub / Twitter

Thanks for reading and stay tuned for more tutorials and updates!