dimafeng.com

neotypes: Akka-http + neo4j

Thu, 27 Dec 2018 00:00:00 +0000

I haven’t been writing posts for a while and this is going to be my first blog post for this year. Today, I’m going to introduce you to my most recent pet project - neotypes.

We’re going to be talking about graph databases today. If you’re not familiar with graph databases and neo4j particularly, please check this article.

I’ve been using neo4j for about a year now and I absolutely love it. Of course, it’s not a silver bullet but it’s something new that helps to model your domain on a completely different level. Since I also love Scala and functional programming, I was missing a “native” Scala driver. So I decided to build it by myself.

neotypes is a Scala lightweight, type-safe, asynchronous driver (not opinionated on side-effect implementation) for neo4j. Let’s break this sentence down into distinct properties of this driver.

Scala - the driver provides you with support for all standard Scala types without the need to convert Scala <-> Java types back and forth and you can easily add your types.
Lightweight - the driver depends on shapeless and neo4j Java driver
Type-safe - the driver leverages typeclasses to derive all needed conversions at the compile time.
Asynchronous - the driver sits on top of asynchronous Java driver.
Not opinionated on side-effect implementation - you can use it with any implementation of side-effects of your chose (scala.Future, cats-effect IO, Monix Task, etc)

Ok, let’s jump to a real example.

Akka-http + neo4j

We will be building a sample application using Akka-http as a web server and neo4j as a database. Akka-http is pretty popular framework in Scala ecosystem for building http applications such as sites, http APIs, microservices, etc. Akka and Akka-http are based on scala.concurrent.Future, so using a neo4j Java driver would be a challenge. What you would do is you would deal with tons of java.util.concurrent.CompletionStage to scala.concurrent.Future conversion and try to map query outputs to your domain classes manually. neotypes offers a better approach.

Our application is going to be a movie catalog based on the Movie graph dataset which comes with neo4j. There is an existing application built by neo4j team as an example of java driver usage. So we can rewrite it to Scala.

Environment configuration

To start, we need a neo4j instance to run our queries against. I’m a docker fan, so we going to use a docker compose to run my local instance of the database.

My docker-compose.yml looks as follows

neo4j:
  image: neo4j:3.5.0
  ports:
   - "7474:7474"
   - "7687:7687"
  volumes:
   - ./db/dbms:/data/dbms

Execute docker-compose up to start the database. After it’s up and running, go to http://localhost:7474/browser/ and execute :play movie-graph in the query window. Now we have the sample data.

If you run a query MATCH p=()-->() RETURN p LIMIT 25, you will see something similar to:

Data access layer

One of the queries that we want to run is to find a movie cast by its title. The query that is used in java application looks as follows:

MATCH (movie:Movie {title: $title})
       OPTIONAL MATCH (movie)<-[r]-(person:Person)
       RETURN movie.title as title, 
              collect({name:person.name, job:head(split(lower(type(r)),'_')), role:head(r.roles)}) as cast
       LIMIT 1

We need to come up with a model that the result of this query will be mapped to. We could use anonymous types such as Tuple or HList but it’s easier to map the result to a case class and then convert it to json object as-is. In our case, we can use these two case classes:

case class Cast(name: String, job: String, role: String)
case class MovieCast(title: String, cast: List[Cast])

Since the query output has aliases for each element, they will be mapped to corresponding fields of a case class.

Now we can write some queries using neotypes. For simplicity, a service layer is incapsulating both business logic and data access logic (do not do it this way in your production code).

import neotypes.Driver
import neotypes.exaple.akkahttp.MovieService.MovieToActor
import neotypes.implicits._

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

class MovieService(driver: Driver[Future]) {

  def findMovie(title: String): Future[Option[MovieCast]] = driver.readSession { session => 
    c"""MATCH (movie:Movie {title: $title})
       OPTIONAL MATCH (movie)<-[r]-(person:Person)
       RETURN movie.title as title, 
              collect({name:person.name, job:head(split(lower(type(r)),'_')), role:head(r.roles)}) as cast
       LIMIT 1""".query[Option[MovieCast]].single(session)
  }

  ...
}

Let’s take a look at this code. In order to access neo4j, we need to obtain an instance of a Driver. For modularity and testing purposes, the driver is injected via the constructor. As you can see, we’re passing an instance of neotypes.Driver parametrized by [Future] which means that we’re using neotypes wrapper for org.neo4j.driver.v1.Driver and all side effects will be encapsulated in scala.concurrent.Futures.

neotypes.Driver has a very handy method readSession (and the same write alternative) for automatic resource management. Using this method, you don’t need to think about session closing as well as transaction commits/rollbacks - it’s all done by the library.

c"""MATCH (movie:Movie {title: $title})... is custom string interpolator. It means that $title is not regular interpolation. neotypes converts all variables used in a query into neo4j supported types and passes values via statement parameters so that the actual query will look like MATCH (movie:Movie {title: $p1})... and the actual title value will be passed as p1 -> [value]

Now we’re coming to the magic part. query[Option[MovieCast]] defines our mapping. As you can see, we haven’t explicitly specified the mapping rules because neotypes can infer those rules for you. Wrapping of your target type into Option helps us to deal with no result cases.

Akka-http

Let’s now define a simple router. Our goal is to handle /movie/[movie name] urls.

class MovieRoute(movieService: MovieService)(implicit executionContext: ExecutionContext) {
 
  val route = pathPrefix("movie") {
    pathPrefix(Segment) { title =>
      pathEndOrSingleSlash {
        get {
          complete(movieService.findMovie(title).map {
            case Some(movie) =>
              OK -> movie.asJson
            case None =>
              BadRequest -> None.asJson
          })
        }
      }
    }
  }
}

it’s pretty straightforward. Since we’re dealing with scala Futures, we don’t need to convert our service layer output, we just map it to a proper http response.

Wrap it up

Now we just need to glue everything together.

val driver = GraphDatabase.driver(config.database.url, AuthTokens.basic(config.database.username, config.database.password)).asScala[Future]

val movieService = new MovieService(driver)
val httpRoute = new MovieRoute(movieService)

Http().bindAndHandle(httpRoute.route, config.http.host, config.http.port)

In the beginning, we create regular org.neo4j.driver.v1.Driver, and then wrap it into neotypes Driver by using .asScala[Future] method. That’s it!

To demonstrate the application, we can use curl to call endpoints.

$ curl http://localhost:9000/movie/The%20Matrix

{"title":"The Matrix","cast":[{"name":"Emil Eifrem","job":"acted","role":"Emil"},{"name":"Joel Silver","job":"produced","role":"null"},{"name":"Lana Wachowski","job":"directed","role":"null"},{"name":"Lilly Wachowski","job":"directed","role":"null"},{"name":"Hugo Weaving","job":"acted","role":"Agent Smith"},{"name":"Laurence Fishburne","job":"acted","role":"Morpheus"},{"name":"Carrie-Anne Moss","job":"acted","role":"Trinity"},{"name":"Keanu Reeves","job":"acted","role":"Neo"}]}%

I also borrowed a UI part of the java sample application so that you can enjoy a nice looking visual representation.

Conclusion

As you could see, neotypes drastically simplifies interaction with neo4j from Scala. If you want to learn more and help the project, please:

Star the project on GitHub - https://github.com/neotypes/neotypes
Read the official documentation - https://neotypes.github.io/neotypes/docs.html
Check the sources of the application we built today - https://github.com/neotypes/examples

Flexible alerting for ELK

Sun, 24 Sep 2017 00:00:00 +0000

For the past month, I’ve been working on my pet project - L3rt. It’s a very flexible alternative to Elastalert. The main idea was to replace yml configurations with Groovy DSL and create some simple web UI to create and debug rules. In this post, I’m going to show you what I’ve got so far.

Disclaimer

This application is under heavy development and can have bugs. Please use it with caution.

Note

For all who is interested in participating in development or have brilliant ideas for the project's future, don't hesitate to submit issues and send pull requests. L3rt

In order to show you how it all works, I’ll be using this preconfigured docker-compose example of ELK stack. In this project, we already have all that we need for monitoring demonstration, the only we need to add is an alerting tool.

Open docker-compose.yml and add the following service:

  l3rt:
    image: dimafeng/l3rt:0.1.5
    ports:
      - "8080:8080"
    networks:
      - elk

That’s it! Now we’re ready to check how it works.

Execute $ docker-compose up -d to start all docker containers. Great, we just need to load some logs to our monitoring system.

$ nc localhost 5000 < /path/to/logfile.log

If you want to make sure all data is there, you can set up Kibana index patterns using this instruction. But I skip this step for now.

Web playgroud

Open http://localhost:8080 in your browser. You should see something like this:

Now we can write some rules. The most basic rule is a query rule. Let’s say, we want to retrieve messages that contain a sentence “Service started”.

rule {
    ruleName = "myTestRule"                         // (1)
    params = [
            index: "logstash-*",                    // (2)
            
            query: [                                // (3)
                query: [
                            bool: [
                                must: [
                                        ["match": [ "message": "Service started" ]]
                                    ]
                            ]
                        ]
                ],
            
            config : [                              // (4)
                    sources: [
                            ["url": "elasticSearch:http://elasticsearch:9200"]
                    ]
            ]
            
    ]

    reaction { messages ->
        log(messages)                               // (5)
    }
}

Let’s take a look at what the rule consists of.

You should name your rule. The name will be used for storing of rule execution metadata.
This is an index that will be used for querying. By default, Logstash is using indexes logstash-*.
It is an ElasticSearch query. You can read more about ElasticSearch search API here
Since we haven’t specified config (yet), we need to define an ElasticSearch connection.
Let’s just print the result as a log message for now.

In my case, I see the following messages:

It prints all messages matched by the query in a raw format. Now we would want to extract message data and send it to some real target. First, say we have a message that looks like 2016-12-10 17:27:01] [info] [ 4844] Service started in 1093 ms. and we want to retrieve an integer value of milliseconds and depending on that value we’ll be sending or ignoring the message.

How to extract that value - since the rule is just a groovy based script we can use all the JVM power.

def regex = "(?s).*Service started in (\\d*).*"
if(it.data.message.matches(regex)) {
    def startTime = it.data.message.replaceAll(regex, "\$1").toInteger()
    ...
}

Now, if the value is greater than 1000 then we send a notification to a DevOps by email. If it’s greater than 2000 we send an email to DevOps and a HipChat message to a public room of our company. Otherwise, we ignore it. It can look as follows:

    reaction { messages ->
        messages.forEach {
            def regex = "(?s).*Service started in (\\d*).*"
            if(it.data.message.matches(regex)) {
                def startTime = it.data.message.replaceAll(regex, "\$1").toInteger()
                if(startTime >= 2000) {
                    email("devops@company.com", "Enormous starting time", "${it.data.get("@timestamp")}: Enormous starting time detected - ${startTime}")
                    hipchat("Notifications", "Enormous starting time detected - ${startTime}", "RED", true)
                } else if (startTime > 1000) {
                    email("devops@company.com", "Starting time warning", "${it.data.get("@timestamp")}: Starting time is out of normal - ${startTime}")
                }
            }
        }
    }

After rule launch you’ll see:

2017-10-04T01:05:56.096Z[INFO] Email: recipient=devops@company.com subject=Enormous starting time body=2017-09-25T00:49:22.831Z: Enormous starting time detected - 2093
2017-10-04T01:05:56.096Z[INFO] HipChat: room=Notifications message=Enormous starting time detected - 2093 color=RED notify=true

Real configuration

L3rt didn’t send real messages, it printed debug information to the log. In order to send real messages, we have to configure targets.

Let’s create a config file ./l3rt/conf.json:

{
  "sources": [
    {"url": "elasticSearch:http://elasticsearch:9200"}
  ],
  "targetSettings": {
    "mailServer": {
      "host": "smtp.gmail.com",
      "port": "465",
      "auth":  true,
      "username": "myemail@test.com",
      "password": "password"
    },
    "hipchat": {
      "accessToken": "...",
      "baseUrl": "https://api.hipchat.com/v2/"
    }
  }
}

And modify compose file as follows:

  l3rt:
    image: dimafeng/l3rt:0.1.5
    ports:
      - "8080:8080"
    networks:
      - elk
    volumes:
      - ./l3rt/conf.json:/l3rt/config/config.json

Now we can restart all services and start from the beginning - load some rules and run the rule. If all have been configured correctly, you’ll get real notifications to your email and hipchat room. You also can remove the config section from the rule declaration since we have it in the permanent configuration file.

Rule execution metadata

The last thing we haven’t accomplished yet is preventing the rule to retrieve the same data each execution. If you take a look at the query, you’ll notice that each time this rule is executed, the same query is generated, hence each 5 seconds you’ll be receiving the same messages. To ignore old results, we have to explicitly specify what ElasticSearch records should not be taken into account. This can be achieved by using lastSeenTimestamp. It’s a variable that is available within the rule which contains the last timestamp seen during the previous rule execution.

The final rule would look like this:

rule {
    ruleName = "myTestRule"
    params = [
            index: "logstash-*",
            
            query: [
                query: [
                            bool: [
                                must: [
                                        ["range": ["@timestamp": ["gt": lastSeenTimestamp]]],
                                        ["match": [ "message": "Service started" ]]
                                    ]
                            ]
                        ]
                ]
            
    ]

    reaction { messages ->
        messages.forEach {
            def regex = "(?s).*Service started in (\\d*).*"
            if(it.data.message.matches(regex)) {
                def startTime = it.data.message.replaceAll(regex, "\$1").toInteger()
                if(startTime >= 2000) {
                    email("devops@company.com", "Enormous starting time", "${it.data.get("@timestamp")}: Enormous starting time detected - ${startTime}")
                    hipchat("Notifications", "Enormous starting time detected - ${startTime}", "RED", true)
                } else if (startTime > 1000) {
                    email("devops@company.com", "Starting time warning", "${it.data.get("@timestamp")}: Starting time is out of normal - ${startTime}")
                }
            }
        }
    }
}

Running the rule in background

Since we have our rule done, we would want to start using it in background. To do so, we just need to save the rule. Click save button and type a name. That’s it! Now the rule will be executed automatically in background by timer (by default, each 5 seconds)

What’s next?

If you practice Infrastructure-As-Code approach, you would probably prefer to mount existing folder with rules to your container. It’s easy to do - just mount it to /l3rt/rules/
Take a look at other rules. For now, I have got just one more - Count Rule
Contribute your fixes or features

You can find the configured docker compose file used in the article in this github project.

Pragmatic view of the modern front-end application development

Sun, 23 Apr 2017 00:00:00 +0000

I wrote several posts about the way I build back-end applications using Scala and today it’s time to show you what modern front-end application may look like from a pragmatic point of view. Why do I call this pragmatic? Because I consider myself as a full-stack engineer so I prefer more mature approaches rather than the bleeding edge.

Disclaimer

This article is intended to be read by full-stack engineers or inexperienced front-end developers. Front-end experts may find this article outdated.

Note

The project described in this article is available on my GitHub profile.

I had been using multiple approaches since I started writing web applications until I came up with the current one. The evolution of techniques I was following was like this:

Pure JS -> JQuery -> Vaadin -> AngularJS -> (current approach)

All of these approaches were big steps in relation to previous ones, but they still had a lot of downsides. The main disadvantage of most of them is a high support cost (this doesn’t apply to Vaadin, but it has its own kinds of problems).

What a modern web application is
Project structure
Project’s stack
Wrapping up

What a modern web application is

The JavaScript world is the most dynamic and fast growing environment and therefore what I’m saying today will be quite outdated tomorrow and absolutely useless in a week. But there are some stable trends which will be relevant for a longer period of time.

Per my understanding, a modern web app today should:

Be a Single Page Application.
Use dependencies (no manual copying of js files to the /js folder, the same is applicable to css).
Have automatic packaging. A resulted js should only have code that is used (not entire node_module with all dependencies).
Allow you to write in latest JS versions (>=ES6) or TypeScript and produce ES5/ES6 compatible code.
Provide you with browser auto-reload in the dev mode.
Conduct type checking.
Allow writing unit tests.

In this article, I’m going to show you how you can achieve all these goals on the example of a real application.

Project structure

Dependency management

The time when it’s ok to copy js files into /js folder has come and gone. Now all libraries you may want to use are stored on npmjs.com. To start working with it, you need to install node and npm or yarn (yarn is an npm alternative that claims that it provides faster and more reliable user experience while preserving npm file/command format).

It used to be popular to use Bower as a client-side application dependency manager, but it now seems Bower is dying so npm/yarn is more preferable way.

The entry point to an npm project is package.json:

{
  "name": "typescript-react-template",
  "version": "1.0.0",
  "description": "",
  "main": "",
  "scripts": {
    "build": ...,
    "dev": ...,
    "lint": ...,
    "test": "jest"
  },
  "dependencies": {
    ...
  },
  "jest": {
    ...
  },
  "devDependencies": {
    ...
  }
}

The dependencies section defines what libraries will be available in the application scope. An actual dependency looks like this "react": "15.4.2".

Note

By default, the '--save' npm flag saves all versions in the caret range syntax. This is a bad practice in the Java world although this is a common practice in the JS world. This may cause random build failures and that’s why I prefer the fixed version notation.

The devDependencies section covers all dependencies used during build/test/etc.

After you have package.json configured, you can execute npm install. This will download all dependencies from npmjs.com and put into the node_modules folder.

TypeScript

Since I wanted to have a more supportable and predictable codebase, I found TypeScript very helpful. TypeScript is a typed superset of JavaScript. It means that you write JavaSript as you did before but you have the ability to optionally specify types in your code, as well as to have all external libraries to be typed. Moreover, since the code is typed, IDEs can do smarter autocompletion and help with refactoring. All type errors will be shown during the transpilation process.

To make it work you need to have the TypeScript compiler in your devDependencies and compiler configuration as follows:

{
  "version": "0.0.0",
  "compilerOptions": {
    "module": "commonjs",
    "target": "es5",
    "jsx": "react",
    "sourceMap": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "noUnusedLocals": true,
    "noUnusedParameters": true,
    "outDir": "dist"
  },
  "typeRoots": ["node_modules/@types"],
  "exclude": [
    "dist",
    "node_modules",
    "build"
  ]
}

This is a typical configuration that transpiles TypeScript to JavaScript (ES5 standard which is supportable by most popular browsers).

In order to use external dependencies in a typed way, you need to find and put a typing file (*.d.ts) inside your project for each dependency that is used from your code. Some libraries have built-in typings, but for those that don’t, there is a repository with community driven typings for hundreds of libraries. All these typings are available in the NPM with a prefix @types/ e.g. typings for react are @types/react so you can install it as follows: npm install --save react @types/react.

If everything is configured correctly, you can compile your project from the command line by using tsc.

Webpack

Now we have all dependencies and our code base and we need to do ‘something’ in order to produce resulting js and css files. There used to be a couple of standard approaches to do that ‘something’ - Grunt and Gulp. Both of them knew where your sources and dependencies are and then were able to concatenate all of it and put to a dist folder.

Then wepback came to the scene. The main idea of it is to have a declarative configuration that knows only where your sources are and is able to understand what dependencies (not only JavaScript) are used in the code. Knowing all of that, webpack can produce more compact resulting bundles with fewer efforts. As a bonus, it has a built-in auto-reload server.

When I say declarative, I mean something like this:

{
    context: __dirname + "/app",
    entry: "./index.js",
    output: {
        path: __dirname + "/dist",
        filename: "bundle.js"
    }
}

And it’s all you need to write to glue all js files starting with ./index.js and going deeper to its dependencies.

Webpack works with loaders. Since we want to write code in TypeScript we need to setup the following loader npm install --save-dev ts-loader and add the following object to the webpack config.

...
              {
                test: /\.(ts|tsx)$/,
                exclude: /(node_modules|bower_components)/,
                loaders: ['react-hot-loader/webpack', 'ts-loader']
              }
...

This loader will be using the configuration we defined in the TypeScript section.

Usually, when you write web applications you would want to have two different scenarios - development and production. A development one should support hot code reload, produce unminified js with source maps and have some dev-specific flags, while a production scenario is intended to be used as a final bundle packager. To achieve this goal, you can create multiple configs with a shared common part. Since Webpack is run on node.js you can treat its configs as regular javascript code so that you create a common part as an exported js object and then merge it with production and development ones using the plugin webpack-merge.

Linting

When you collaborate with other people, you want to keep your code style consistent. Also, it’s good to have some automated tool that can check common anti-patterns and errors. All these checks are provided by tslint. It can be installed from NPM as follows npm install --save-dev tslint.

There are a bunch of built-in rules that you can work with out of the box. To start using it, you just need a config file that may look like this:

{
  "rules": {
    "class-name": true,
    "comment-format": [true, "check-space"],
    "indent": [true, "spaces"],
    "no-duplicate-variable": true,
    "no-eval": true,
    ...
    "jsx-self-close": true
  },
  "rulesDirectory": [
    "node_modules/tslint-react/rules"
  ]
}

Now you can run tslint checks within your codebase tslint "src/**/*.ts*" (to make tslint executable you need to install it globally). This already is quite good but we’ll configure it to not execute this command manually. This can be done by webpack’s loader. In order to do it, you need to install tslint-loader as a dev dependency and configure it in the webpack configuration:

{
    preLoaders: [
            {
                test: /\.ts[x]*$/,
                loader: "tslint"
            }
        ],
  ...
}

Now, each compile process will be prepended by tslint check and you’ll see all errors and warnings as soon as you save the source file.

Build steps

The package.json is quite limited in imperative task declaration - you can’t specify complex logic of your build steps. The only you can do is define commands that should be executed. It’s not powerfull but at least you don’t need to remember all application lifecycle commands.

Usually, I have these four scripts:

"scripts": {
  "build": "NODE_ENV=production webpack -p --config webpack.production.config.js --progress --profile --colors",
  "dev": "webpack-dev-server --progress --profile --colors --hot",
  "lint": "tslint \"src/**/*.ts*\"",
  "test": "jest"
}

build produces production-ready bundles that you can distribute as is.
dev starts dev server.
lint checks the codebase using linters.
test launches unit tests.

To start any of the scripts, you can run npm with the following arguments npm run [script name]

Project’s stack

One year ago I wrote a post about om (clojurescrit library based on react.js). Now it’s time to bring React.js to a real life project in the enterprise-ish world.

React.js

I think there’s no need to describe what React.js is. This framework if pretty popular right now and has a lot of hype around it. TypeScript fully supports JSX so you can write React components and benefit from type checking. A typical component may look like this:

interface Props {
    userId: string;
}

export class UserView extends React.Component<Props, any> {
    ...
}

Now if you try to use it with an incorrect type (e.g. <UserView userId={1} />) you would get a type check error.

Communication with a backend API

I liked the idea of using rx.js instead of promises. An HTTP client on top of rx.js was introduced by angular 2 and I found it very useful. It looks like you work with promises when you have simple scenarios and brings all the power when you need to deal with dependent requests, delayed execution, execution timeouts, etc.

I use a library called ts-rx-rest which I contribute to. Basically, it’s a typed HTTP client with an rx.js interface and it does’t make you follow any particular approach – you can use it in any typescript project. Usually I put all HTTP related code to separate stateless *Service classes so it can be mocked in unit tests.

export class UserService {
  getUsers(): Observable<Array<User>> {
    return Rest.doGet<Array<User>>(Config.API_URL + '/users');
  }
  ...
}

Now let’s assume you need to load a user profile with permission list for his group. If you’re waiting too long for any of these two requests then we want just to show an error.

const user$ = userService.getUser(userId);
const userPermissions$ = user$.flatMap(user => permissionsService.getPermissions(user.groupId));

Observable.zip(user$, userPermissions$)
  .timeout(200)
  .subscribe(
      ([user, userPermissions]) => {
        // process 'user' and 'userPermissions'
      },
      e => {
        // process errors
      }
  )

This looks very clear, doesn’t it?

MobX

When I just started using React.js I was trying to use pure components and manage their state manually via setState but this approach is not scalable. Then I found redux, redux looks very promising? but there’s one big problem - it’s way too verbose. In the meanwhile, my co-worker had found MobX and it had become a relief.

Now I split all TypeScript code into three types of components:

React component - JSX component with no state management
Store - component’s state
Service - stateless HTTP API layer

React components are only responsible for rendering – they know nothing about the state. All state management is located in stores as well as event reactions (e.g. button clicks, input value changes, etc).

MobX introduces two main roles of components: observer and observable. An observer is an object whose state changes should be tracked by observers. In our case, a react.js component is an observer that is re-rendered once a state of the observables (only those that are used in the render() method) has been updated. It may look like this:

export class UserViewStore {
    @observable users: User[] = [];

    //...
}

...

@observer
export class UsersView extends React.Component<Props,any> {

  //...

  render() {
    return <div>{store.users.map(u => <p>{u.name}</p>)}</div>;
  }
}

When you change users by adding/removing a value to/from the array or assign a new array to the variable, UsersView will be re-rendered automatically.

Note

If you need to update multiple variables consequently or do multiple changes to an array, you need to think of atomicity of the re-rendering because each update will cause virtual DOM recalculation. In these cases, you need to use the runInAction wrapper function.

Dependency Injection

MobX provides you with build-in Dependency Injection functionality but it only works with react.js components. When you have multiple stores which depend on each other, you would like to have an ability to inject stores to other stores as well as to inject stores to react.js components. I haven’t found any libraries that solve this problem so that I wrote my own solution:

export const diContext: any = {};

export function diInject(): any {
    return function (target: any, propertyKey: string) {
        Object.defineProperty(target, propertyKey, {
            get() {
                const bean = diContext[propertyKey];
                if (bean) {
                    return bean;
                } else {
                    throw new Error(`Context has no bean with name ${propertyKey}.
                    Available beans: ${Object.getOwnPropertyNames(diContext).join(', ')}`);
                }
            },
            set() {
                throw new Error('Not allowed');
            },
            enumerable: true,
            configurable: true
        });
    };
};

This injection decorator uses globally defined diContext and is able to dynamically inject objects from diContext by name.

diContext.userService = new UserService();
...
export class UserViewStore {
    @diInject() userService: UserService;
    ...
}

Testing

You can’t deliver reliable software without testing nowadays. It used to be extremely difficult to test javascript but now you can run unit tests in node.js or in a browser very easy. I prefer using jest. The main advantage is that jest is a Zero configuration testing platform – it just works, moreover it works with TypeScript.

To start, we need to install jest npm install --save-dev jest-cli @types/jest, then add this small config object to package.json

"jest": {
    "moduleFileExtensions": [
      "ts",
      "tsx",
      "js"
    ],
    "testRegex": "/__tests__/.*\\.(ts|tsx|js)$"
  }

Now ready to go!

Beside jest, I am going to use typemoq, a mocking library for typescript: npm install --save-dev typemoq. Let’s look at some tests. Per our configuration, tests should reside in ./__tests__/. Here is an example of a test that works with a store.

it('UserViewStore should be filled with data', () => {
    const user: User = {
        id: 1,
        firstName: "test",
        lastName: "test"
    };

    const userServiceMock = TypeMoq.Mock.ofType(UserService);
    userServiceMock.setup(x => x.getUsers()).returns(() => Observable.just([user]));

    diContext.userService = userServiceMock.object;
    const store = new UserViewStore();

    expect(store.users.length).toEqual(1);
    expect(store.users[0].firstName).toEqual("test");
});

As you can see, we prepared a service layer so that it doesn’t make a request to the real backend. Instead, it works with Observable.just that just executes its body right away. After that, we validate the state of the store with a built-in chain expect(...).toEqual(...).

That looks great, doesn’t it? Want more? Due to a functional nature of react, we can easily test JSX output. Since we don’t use react component’s state, we just need to explicitly define stores’ state. Then we can calculate the virtual DOM and compare it with a snapshot. The snapshot creation and comparison is a built-in feature of jest.

it('UsersView should render proper DOM', () => {
    const user1: User = {
        id: 1,
        firstName: "test",
        lastName: "test"
    };
    const user2: User = {
        id: 2,
        firstName: "test2",
        lastName: "test2"
    };

    const userStoreMock = TypeMoq.Mock.ofType<UserStore>();
    const userViewStoreMock = TypeMoq.Mock.ofType<UserViewStore>();
    userStoreMock.setup(x => x.createUserViewStore()).returns(() => userViewStoreMock.object);
    userViewStoreMock.setup(x => x.users).returns(() => [user1, user2]);

    diContext.userStore = userStoreMock.object;

    expect(create(<Router><UsersView/></Router>).toJSON()).toMatchSnapshot();
});

The first run of the test (npm run test) will generate a snapshot, that will be treated as the correct value, further runs will just compare produced DOM with a snapshot.

Dev Tools

When you write real applications at work you would like to be productive as more as you can. Fortunately, there are multiple tools can help to achieve it.

Since we’re using webpack, there is a built-in browser page refresher. All you need to do it to run npm run dev. After each change, your page will be auto-refreshed.

The next tool that saves your life is Google Chrome Dev Tools and plugins. First of all, Dev Tools simplifies life by providing DOM navigator, network monitoring tools, debugger, profiler, etc. You can debug your TypeScript code without any additional extensions - it just works. If you need more advanced tools, you can take a look at React Developer Tools. It is a google chrome extension that can show a virtual DOM with all its attributes. Sometimes, it’s very helpful to check what is the current virtual DOM and how it relates to real DOM.

Another helpful extension is MobX Developer Tools. This extension is toolset that exposes MobX’s magic. It can show DOM re-rendering, actions and reactions, observable changes, etc.

Wrapping up

There is no one proper way to create applications in the JS world. The JS environment is extremely dynamic – every day you can find a new library that simplifies or improves some aspects of application development. I described my current approach and I have to doubt that in several months 50% of described techniques will be rethought and it will a new article in my blog.

5 drawbacks to liking SBT

Wed, 04 Jan 2017 00:00:00 +0000

I have an opensource project written in scala called testcontainers-scala (if your heart beats faster when you hear docker, be sure to click on the link). Several days ago I received a pull request that adds scala 2.12 support to my project. Sounds great, right? But unfortunately that broke scala 2.11’s compatibility. To solve this issue, there is a quite standard solution in the scala world - Cross-building. I hadn’t had much experience with SBT by that time and I had decided to give a try. In this post I’m going to show you what came of this.

In the comments on Reddit for my article about integration testing in scala, somebody said:

Even though I “use” sbt I wouldn’t exactly say I know to use it. All I do is copy and paste crap for the most part.

I spent two days trying to replicate the functionality of my Gradle script using this “copy and paste crap” approach. I briefly read the official documentation, reviewed a lot of stackoverflow threads and after all I lost hope. Here’s why:

1. Speed

SBT is super slow. Each time I run tests there were several preparation steps that were downloading and checking some random things. Subjectively, everything took at least 1.5 times more time than it took in the Gradle script (especially using the Gradle daemon).

2. Project properties

My gradle script reads properties from file and It’s even a built-in Gradle feature. I have two properties files: a local one that is stored along with the project and a global one located in ~/.gradle. If I want to override some property for a particular run, I can use -D and -P command line options.

I needed the same in SBT. I googled it - this is what I found.

import java.util.Properties

val appProperties = settingKey[Properties]("The application properties")

appProperties := {
  val prop = new Properties()
  IO.load(prop, new File("application.properties"))
  prop
}

name := appProperties.value.getProperty("name")

Really? What do I do if I want to have both a local file for versions and a global one for my private data? What do I do if I want to override it via a command line option? - Code it!

3. Test exclusion

I have two type of tests: unit and integration. Because the code base of the project is really small, I keep all of them in one sourceset src/test/scala (I know - I know, it’s not a good practice). But I cannot run integration tests every time I start a test task because Travis-ci doesn’t allow me to run docker in my tests. In my gradle script, I do a simple trick:

test {
    exclude 'com/dimafeng/testcontainers/integration/**'
}

task integrationTest(type: Test) {
    include 'com/dimafeng/testcontainers/integration/**'
}

It’s very simple and clear. If I had a bigger codebase I could also do:

Add an extra sourceset
Create a sub-module

What do we have in SBT? - Tags. Ok, I found some examples here and here. Also, we need to modify a task test to make it exclude these tests and create a new one which works with only these. Here’s the documentation.

Wait, I’m lost, how to glue all of the pieces together? Where to find the missing ones?

Maybe I can configure an additional sourceset or create a sub-module? - Yes, but you’ll have to read the entire “spaceship manual”.

4. Publishing to maven central

My library should be published to maven central to be accessible by all build tools. I found a documentation - Publishing.

There are two ways to specify credentials for such a repository. The first is to specify them inline: … The second and better way is to load them from a file, for example:

credentials += Credentials(Path.userHome / ".ivy2" / ".credentials")

Wait, I thought there are no built-in tools to work with properties files. Oh, that works for credentials only.

I think I can live with that. What next? I need to sign my binaries. So here’s my signing key. Where to put it?

Note: While it is general practice to drop the higher-order bits of 64-bit integer keys when passing ids around, the PGP plugin requires the full key id currently.

Not a problem - gpg --keyid-format LONG -k

usePgpKeyHex("7cf8d72be29df322")

But I want to read my key from the properties file like this usePgpKeyHex(appProperties.value.getProperty("signing.password")).

build.sbt:82: error: `value` can only be used within a task or setting macro, such as :=, +=, ++=, Def.task, or Def.setting.
usePgpKeyHex(appProperties.value.getProperty("signing.password"))
                           ^
[error] Type error in expression
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?

5. Releasing

I don’t like to keep complex manual procedures in mind - I automate everything. In Gradle, I have a release plugin. It’s very simple and flexible. What it does:

Asks what version to release and updates the current version in the project properties
Tags the current git commit
Pushes the commits to upstream
Using gradle task dependencies, builds project, runs tests, uploads archives

$ ./gradlw release

That’s it. It can’t be easier. What we have in SBT? - Painless release with SBT

Wait, do I need to write more code than I have in my libriry just to release? I’m not even sure that it can be blended with code I wrote on the previous steps - I don’t even want to try otherwise I’ll have to be debugging it for new two days. Let’s leave it in the old-fashioned way - manually.

Conclusion

After two days I gave up. I didn’t find any features in SBT which I don’t have in Gradle. Afterwards, I implemented cross-building in my gragle scripts. It’s less elegant but works well and doesn’t break all what I had before.

As for SBT, It seems I’m not the only one who struggles with it. I think, a build tool shouldn’t be so complex and counter-intuitive. The lack of good documentation and good examples makes the quick start quite impossible. You can just sit and copy and paste from documentation and have something working in a few hours. All you need to do is dig and pray so that in 10 years you’ll be lucky to get a “Ph.D. in SBT”. The current state of SBT doesn’t look mature, all standard procedures which are built-in to maven or Gradle should be coded by yourself.

So, I’ll stick with Gradle.

Distributed systems cheat sheet

Sun, 04 Dec 2016 00:00:00 +0000

Since last year, I’ve been dealing with multiple distributed systems and suffering from a lack of structured understanding of how it all work under the hood. This post aims to break the subject down into small parts and aggregate multiple sources and links I found useful. It covers mostly distributed data storage systems but some concepts may be applied to other types of systems.

Disclaimer

I'm not an expert in distributed systems. This blog post reflects my current understanding of the subject. If you find any mistakes, you're welcome to leave comments.

What a distributed system is

A distributed system is a software working on multiple computers/VMs/containers (further in this article I’ll call all of this a node) and communicating with each other via network interfaces. It helps solving scalability and reliability problems by using the following techniques:

Data replication is copying of data to multiple nodes.
Data partitioning is when data is divided into N parts and processed separately on several nodes.

Let’s dive into this more deeper:

Physical factors
Time and Order
CAP
Consistency
Replication
- Synchronicity
- Node relationship
  - Master/Slave (single copy)
  - Multi-master
Consensus
Partitioning
- Horizontal partitioning
Links

##Problems

There are two different worlds - theoretical model and practical experience. They both don’t meet each other in most cases because these models describe a perfect environment, but there is an infinite number of problems that can cause a divergence.

Physical factors

All nodes are connected to each other via the network and due to the many factors, this doesn’t provide strong guarantees of data transferring and accessibility. Fallacies of distributed computing explains it as follows: Network is not reliable. Packages may be lost, reordered, or received multiple times, with different non-zero delays and limited throughput of the network. Moreover, the topology of the network is not constant - it can be changed at any time intentionally or accidentally due to network failures, also nodes can crash or be rebooted.

Time and Order

When we have a sequence of events occurred in a distributed system, it’s important to know in what order they came. We can’t rely on physical clocks because it’s always not perfectly synchronized. That’s why people came up with logical clocks.

Logical clocks are not about actual time, it’s about an order of events, it’s also called a partial ordering. In this model, all events are connected by happens-before relation. There are two implementations of this model:

Lamport timestamps
Vector Clocks

Here is a more formal description of this concept.

CAP

All problems described in previous two sections can be combined in one model called The CAP theorem. It defines the relation between three properties:

Consistency, guarantees that all nodes will see the same data representation;
Availability, guarantees the fastest availability of the data ignoring its accuracy;
Partition tolerance, the system survives network failures.

The theorem states that only two of these three can be chosen in the system. Knowing that the network with no failures and delays doesn’t exist, we have only two types of systems to choose: CP and AP. Although in the real world, popular distributed systems are neither CA nor AP because they can provide different guarantees on demand or weaker consistency guaranties or even loose availability/consistency under certain conditions. This brings us to the idea that the CAP theorem in most cases is a high-level abstraction that can’t be applied to the systems you write or use but can be a starting point for understanding more complex cases.

Consistency

It’s important to know that the Consistency in the CAP theorem means strong consistency, specifically linearizability. But there are many different types of consistency.

Strong consistency all nodes see the values in the same order
- Strict consistency: (impossible to implement in a distributed system) It orders all writes and reads relying on absolute global time.
- Linearizability: the order of reads on all nodes matches the order of writes. A read can obtain the most recent applied value or a value that goes after it.
- Sequential consistency: the order of reads doesn’t match the order of writes but is the same for all read operations.
- Causal consistency: the order of reads of causally-related writes matches the writes order.
Weak consistency the order of reads can differ
- Client-centric consistency: the client works with the distributed system through a cache or uses a session. Unless client writes then reads either through the same cache or session bound to the same node, all reads will be consistent, while other nodes can see an inconsistent state.
- Eventual consistency: “It defines that if no update takes a very long time, all replicas eventually become consistent”

Here are useful slides about consistency models.

Replication

Now we can take a look at more practical aspects of distributed systems. Let’s assume we have data stored in a system working on one node, now we want to have:

fault-tolerant: if one node is down, another one can replace it
higher accessibility: if we have too many reads and clients are experiencing difficulties under high load, then we copy data to another node to spread the load

Both of these goals can be achieved by multiple approaches depending on more detailed requirements.

Synchronicity

The actual copying of the data during the replication can be conducted in several manners.

Synchronous

When a client writes some data to the master node, the node accepts data and saves it to its own storage plus sends this data to all replicas. Once all replicas responded back with OK, the master node returns a response to the client. If data didn’t do through to any of replicas, the client will receive an error code - this guarantees data consistency.

Since the master should wait the response from all of the nodes, a single write operation cannot be faster than the time of the slowest combination of data write to a node + network latency for this node

When one of the nodes becomes inaccessible, the master stops accepting writes - all nodes are only available for reads.

Asynchronous

In this replication approach, a client receives response as soon as master thinks that the data has been handled by the master, the data copying to replicas is starting asynchronously. The client won’t be notified about copying failure. Asynchronous replication offers fewer guarantees but more faster writes from the client point of view.

If the master goes down and some data hasn’t been properly delivered to the slaves, this data will be lost even though the client is aware of data write.

Semi-synchronous

There is a way to combine synchronous and asynchronous approaches called semi-synchronous replication. This approach works the same way as synchronous but returns response once at least one replica acknowledged.

In case of single replica, this approach is equal to synchronous.

Node relationship

Master/Slave (single copy)

The most standard replication model is master/slave replication. In this technique, there should be one master node and single or multiple slaves. Master accepts writes and reads but slaves accept only reads. When master goes down, replicas stay available for reads.

Usually master sends operation log (binary log) or a log with global transaction identifiers, this allows all replicas understand where they are at in that log and catch master up after a failure or initially.

Most real-world systems promote a slave to a master when the master becomes inaccessible.

Multi-master

Multi-master is where you can write on any node and data gets replicated to all others. This addresses the problem of the many writes in HA systems.

Usually, this approach causes different issues due to the fact that clients can work with the same data but from different masters. The following techniques can solve this problem:

Pessimistic locking - master locks a potentially affected segment of data so others master refuse modification of the segment.
Optimistic locking - based on data versioning. If a master A tries to write a value with a version that doesn’t equal to the one that a master B has, this write will be refused.
Conflict resolution
- Priority - from two conflicting writes we apply a one by some rule (e.g. the write with the latest timestamp)
- Data merging - convergent or commutative replicated data type

Conflict avoidance - data is divided into logical parts and assigned to specific master, so clients are aware of what type of data should go to a specific master. This approach is very similar to partitioning.

Consensus

The consensus problem is a problem of having nodes in distributed systems agreed on some value.

Consensus typically arises in the context of replicated state machines, a general approach to building fault-tolerant systems. Each server has a state machine and a log. The state machine is the component that we want to make fault-tolerant, such as a hash table. It will appear to clients that they are interacting with a single, reliable state machine, even if a minority of the servers in the cluster fail. Each state machine takes as input commands from its log. In our hash table example, the log would include commands like set x to 3. A consensus algorithm is used to agree on the commands in the servers’ logs. The consensus algorithm must ensure that if any state machine applies set x to 3 as the n^th command, no other state machine will ever apply a different n^th command. As a result, each state machine processes the same series of commands and thus produces the same series of results and arrives at the same series of states.

This may be illustrated as follows:

Two-Phase commit

The Two-Phase commit algorithm introduces two roles:

Leader (coordinator, master) - a node that a client writes to.
Acceptor (cohort, replica) - a node that all writes should be replicated to.

This algorithm is very simple and relies on the assumption that network is reliable. There are one leader and one or multiple acceptors in the system. When a leader receives the data to write, it proposes the data to all acceptor and waits for a vote from all of them. An acceptor can accept or reject the value base on acceptor’s state, then:

If all acceptors accepted the value, the leader sends a commit request.
If one or more of the acceptors failed, the leader sends a rollback request.

After that, the leader receives acknowledgment for all commit/rollback command and returns to the client. More detailed description of the steps can be found here.

This approach has a critical disadvantage - it’s not partition tolerant and it’s blocking. If the leader fails in the middle of the agreement, the acceptors will be infinitely blocked on waiting for commit/rollback request. If some of the replicas fail permanently, then master won’t be able to accept any writes.

Three-Phase commit

The Three-Phase commit algorithm is an improved version of Two-Phase commit, that aims to eliminate its blocking nature. The algorithm introduces additional revertible phase of pre-commit where acceptor can store the data and switch to the state of waiting for a commit request. If an acceptor doesn’t receive a commit request within specified timeout, the data will be discarded. Even though it solves the problem of blocking, the system can get in an inconsistent state.

Paxos

Paxos is the next generation of consensus algorithms. It solves the weaknesses of the Two-Phase commit and Three-Phase commit algorithms. The main differences are that Paxos requires the majority of nodes to acknowledge in order to send a commit request, in addition to that, the leader assigns a unique sequence number to each write so that an acceptor can distinguish which value comes first.

You may look at a more detailed description here or here.

There are also multiple modifications of the original paxos algorithm like Multi-Paxos, Cheap Paxos, or Fast Paxos.

RAFT

Nevertheless, Paxos is not the only partition tolerant algorithm. Furthermore, Paxos is blamed for its complexity. The Raft algorithm is the youngest from all described above. It’s relatively simple and give the same guarantees as Paxos does.

The best entry point to the details of this algorithms is this site.

Partitioning

Partitioning is a technique of dividing the monolith data into several parts and storing them on different nodes. This technique helps to deal with large data that doesn’t fit one node or deal with a big number of writes/reads when one node of the system can’t handle it. Sometimes it can be used for security purposes (if data should be isolated in a certain way) or when geographical location of data parts is important.

There are three strategies of partitioning:

Horizontal partitioning (sharding) - having a set of objects, each object depending on some rule is stored on the specific node.
Vertical partitioning - having the same set of objects, each node stores only specific attributes of each object (e.g. user name is held by node A while user DOB is held by node B).
Functional partitioning - this strategy divides the data by type (e.g. user info and user orders are located on different nodes).

All these strategies solve a particular problem but bring other problems. The main problems are:

Probable inconsistency. Data on different nodes are not linked so that we can lose some data.
More complex and time-consuming data querying. If the data we need is located on different nodes, we need two steps to obtain the result: query all nodes and then merge the results.

Horizontal partitioning

Horizontal partitioning is the most popular strategy due to the fact that in most cases distributed systems deal with a big number of homogenous objects. Also, this approach gives more scaling flexibility.

The most interesting part of this strategy is the rule that will be used for data division. This rule is called “sharding key”. It should be chosen beforehand (because it’s relatively difficult in most cases to change this key after the system is in use) and should:

Divide data into approximately the same parts so that there won’t be nodes with too much data and with too little data.
Divide data into parts with approximately the same load. It means that hot data shouldn’t be concentrated on a certain node while unused data on another node.
Divide by geography. Data related to specific location should preferably be located in the corresponding geographic data-center.
Data of a node A shouldn’t avoid relations to data of other nodes.

A sharding key can meet one or more of the properties below depending on the problem. There are several common solutions to choose a sharding key:

Range partitioning, the key will be the interval of the value of some sequential attribute (e.g. date of the transaction, user DOB, etc).
List partitioning, the attribute value itself (e.g. user country).
Hash partitioning, the key is calculated by applying a hash function to any of model’s attributes.
Dynamic partitioning, keys are associated with values be a certain rule and these relations are stored in an external system.

How to manage your dotfiles

Sun, 09 Oct 2016 00:00:00 +0000

A few months ago I wrote a post with Linux commands cheat sheet. But then I started thinking that I don’t need to remember all those commands if I have a collection of scripts and aliases which I can use and transfer from one machine to another with ease. Moreover, if had such repository I would store scripts which make freshly installed OS X/MacOs a fully configured development environment. To achieve this goal, people come up with an idea of storing their dotfiles and other useful scripts in some (semi-)standard way on github.

What are dotfiles and (semi-)standard way

dotfiles are files whose names start with a dot. In this post, we’re talking about dotfiles located in ~/. Most of them are configurations of different parts of your system.
(semi-)standard is the way these files are stored on github and there’s no one standard way to do it. There is a site dotfiles.github.io where all good practices are collected on. Basically, you need to structure your git repo in a way that is easy to install and easy to update when some changes pushed to upstream.

In most cases, you need to clone the dotfiles repo to ~/dotfiles and then run an install script. The install script usually backs up old files and creates symbolic links to those from the cloned repo. Besides, a dotfiles repo may contain scripts for installing additional utils and applications and it can make a fresh system a ready-to-go system in minutes.

Which dotfiles to work with

Now we can loot at some files which you can modify to configure system for your needs. I have my own dotfiles (they’re not quite done yet, but I’m working on it) you can use it as an idea. The intention of having your own dotfiles (and not using existing ones) was to understand how it all works and why you need each line in your configs. Basically, my dotfiles is a compilation of all popular dotfiles repositories of which parts I found useful with several modifications.

install.sh

This is an entry point to the dotfiles repo. Currently, my script is split into three parts: config linking, common parts installation and OS specific application installation. The linking step just creates symbolic links to files from the repo. So you can pull new changes and they automatically will be applied. To make work on my dotfiles more smooth, I use names without dots but create symlinks with dots.

install.sh link

Then you can install common application and scripts. Currently, it contains:

oh-my-zsh and theme

Oh-My-Zsh is an open source, community-driven framework for managing your ZSH configuration. It comes bundled with a ton of helpful functions, helpers, plugins, themes, and a few things that make you shout…

Some python packages

install.sh install_commons

The OS specific application installation, in my case, uses brew to install all essential applications. You can review and modify the list of applications in brew.sh and then run:

install.sh install_app_mac

.bash_profile

This is the first file to be loaded when a bash shell is starting. Common practice is to have less code here but load all that you need from separate files grouped by purpose. In my case, I load .aliases and .functions.

.zshrc

This is an entry point to the zsh configuration that loads all others configurations. Since I’m using oh-my-zsh this configuration is from the standard oh-my-zsh distribution with a few changes. You can easily go through the file and uncomment lines you need. In my case, I made the following changes:

Theme ZSH_THEME="dracula" - this is nice looking dracula theme (downloaded during install.sh install)
Plugins plugins=(docker gradle) - here you can specify which plugins should be run.

Also, this config script loads .bash_profile mentioned above, so all aliases and functions will be working in both bash and zsh.

.aliases

All aliases are stored here. The quick overview of several popular repositories gave a basic pattern of this file. Usually, people make their own shortcuts to most used paths in the everyday routine:

alias d="cd ~/Documents/Dropbox"
alias dl="cd ~/Downloads"
alias dt="cd ~/Desktop"
alias dev="cd ~/Documents/dev"
alias g="git"
alias h="history"

.functions

This file contains functions that can be used as regular command line applications. For the convenience, I organized them in the way it could be easily transformed into a documentation.

function mkd() # Create a new directory and enter it
{
	mkdir -p "$@" && cd "$_";
}

Then I added an alias alias dhelp="grep "^function" ~/dotfiles/functions | sed 's/function/+/g' | sed 's/() #/ - /g'" and now it works as follows:

$ dhelp
+ calc -  Simple calculator
+ mkd -  Create a new directory and enter it
+ cdf -  Change working directory to the top-most Finder window location
+ fs -  Determine size of a file or total size of a directory
+ dataurl -  Create a data URL from a file
+ server -  Start an HTTP server from a directory, optionally specifying the port
+ gz -  Compare original and gzipped file size
+ json -  Usage: `json '{"foo":42}'` or `echo '{"foo":42}' | json`
+ digga -  Run `dig` and display the most useful info
+ escape -  UTF-8-encode a string of Unicode symbols
+ unidecode -  Decode \x{ABCD}-style Unicode escape sequences
+ codepoint -  Get a character’s Unicode code point
+ a -  `a` with no arguments opens the current directory in Atom Editor, otherwise opens the given location
+ v -  `v` with no arguments opens the current directory in Vim, otherwise opens the given location
+ o - `o` with no arguments opens the current directory, otherwise opens the given location
+ tre -  `tre` is a shorthand for `tree` with hidden files and color enabled
+ grr -  Git: Revert changes & Remove all untracked
+ gr -  Git: Revert changes in specific file
+ gacp -  Git: add all, commit and push. [-m "comment"]

.gitconfig

If you work with multiple git accounts (e.g. you have github account and git account at your work) here’s the trick - if you have the following lines in .gitconfig

[user]
	name = Dmitry Fedosov
	email = "(none)"

you’ll be asked to specify local email for each repo.

.osx

OS X/MacOs has a built-in command line tool for configuration - defaults. To get an idea which settings to tune, here’s a script that interactively asks you what to change. You can go through it and copy settings into your config.

.vimrc

I wrote a post some time ago about vim configuration. I haven’t changed too much since that time. I just configured nice looking theme Dracula. It’s installed from Vundle, so you just need to run :PluginInstall in your vim.

Integration testing using docker and scala

Mon, 01 Aug 2016 00:00:00 +0000

A few weeks ago I published a post on Reddit where I tried to advertise my lib - testcontainers-scala. This failed, I got no feedback at all. This prompted me to write this post and show how awesome integration testing might be.

Here I’m going to show a simple example of selenium tests. Let’s get it started.

Testcontainers

Testcontainers-scala is a wrapper for testcontainers-java that allows using docker containers for functional/integration testing.

TestContainers is a Java 8 library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

Assumptions

~~You know how to use SBT~~ You’re able to copy and paste dependencies to your build.sbt
You use scala 2.11
You run all of these on java >8
You have docker installed on your machine

If all of those points are true then we’re ready to write some code.

Build script

I’m not going to make it complex, so here’s the simplest minimalistic build.sbt that we need to start. It’s enough for showing the power of testcontainers.

name := "testcontainers-example"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "com.dimafeng" % "testcontainers-scala" % "0.2.0" % "test" //testcontainers itself
libraryDependencies += "org.scalatest" % "scalatest_2.11" % "2.2.6" % "test"
libraryDependencies += "org.testcontainers" % "selenium" % "1.1.0" % "test" // testcontainer-java module for selenium
libraryDependencies += "org.seleniumhq.selenium" % "selenium-java" % "2.53.0" % "test"
libraryDependencies += "org.slf4j" % "slf4j-simple" % "1.7.21" // logger implementation

This is pretty standard build script, so there’s no problem just to copy and paste this into build.sbt of your existing project.

Important note

The version of selenium-java links to a version of selenium/standalone-chrome-debug container. If you are using a version different from 2.53.0 please check a version of the container manually. See the official docker hub page

Specs

Now we can write some specs. Since I have no prepared backend, I’m going to write specs in terms of scalatest which test reddit.com.

package com.dimafeng.scala.example.testcontainers

import java.io.File

import com.dimafeng.testcontainers.SeleniumTestContainerSuite
import org.openqa.selenium.remote.DesiredCapabilities
import org.scalatest.FlatSpec
import org.scalatest.selenium.WebBrowser
import org.testcontainers.containers.BrowserWebDriverContainer.VncRecordingMode


class RedditSpec extends FlatSpec with SeleniumTestContainerSuite with WebBrowser {
  override def desiredCapabilities = DesiredCapabilities.chrome() // (1)
  override def recordingMode = (VncRecordingMode.RECORD_FAILING, new File("./")) // (2)

  // (3)
  "Reddit" should "show non-zero number of links on the main page" in {
    go to "https://www.reddit.com/"

    assert(cssSelector("#siteTable .thing").findAllElements.nonEmpty)
  }

  // (4)
  "Reddit" should "redirect to https" in {
    go to "http://www.reddit.com/"

    assert(currentUrl startsWith "https://www.reddit.com")
  }

  // (5)
  "Reddit" should "stay reddit after click on a random link" in {
    go to "https://www.reddit.com/"

    click on cssSelector(".entry a.title").findAllElements.filter(_.isDisplayed).next()

    println(currentUrl)
    assert(currentUrl startsWith "https://www.reddit.com")
  }
}

This is all you need to test your GUI inside the docker. Now let’s break it down. We have SeleniumTestContainerSuite trait that does all the magic under the hood. It starts a clean new container on each test case, so you don’t need to worry about the state - no data shared between test cases. To make it work in a basic scenario you just need to configure DesiredCapabilities (1) - here it’s chrome, but you can use firefox as well.

There are three test cases (3-5). The first two (3-4) are regular selenium tests wrapped with nice scalatest DSL. And the last one is a test that sometimes fails. But you can say that there is no way to figure out what is going on inside the container and this may be true, but not in our case. As you see in (3) we ask testcontainers to record all failed tests and produce .flv file. This is how it looks like:

The tests case checks that after clicking on a first link in the list a user won’t be redirected to a foreign domain, in fact, he will.

Conclusion

I had a lot of problems with old fashioned approaches like Chrome/Firefox/Remote/etc. drivers running them locally and on CI servers, and I had a tough time with modern approaches like phantom.js. This seems much more reliable and convenient because:

Tests are reproducible: if it fails on CI server it fails on your dev environment and visa verse.
Each test case is stateless: you don’t need to worry about cleaning cookies, local storage, etc. Each test works with its own clean browser.
Easy to start in parallel: since there’s no state, you can run it in parallel.
Simple troubleshooting: if a test fails you can watch recorded video or fallback to local Chrome/Firefox drivers.

It’s free and open-sourced - github project with all instructions. Your contributions are very welcome.

Publishing of your artifact to maven central

Sun, 26 Jun 2016 00:00:00 +0000

Recently, I finished my pet-project and wanted to publish it to maven central. As I found out later, publishing is not a trivial thing. In this post, I’m going to show you all steps which you need to go through to publish your artifact.

My project is a scala wrapper for a very useful library testcontainers-java. As you may assume, it’s written in scala, but all the following is applicable for java/groovy/whatever language project if it’s built by gradle.

Preparation

First of all, you need to request a group id reservation. To do so, you need to sign up in Sonatype’s JIRA and create an issue where you’ll have to ask to approve your group id. This is how I did that.

GPG Keys

The next step is a creation of a gpg key. You need to install some additional utilities, on macos it looks like that brew install gpg gpg2.

Now, tet’s create a key.

gpg --gen-key

Pick default answers to the first two questions. Then enter two years as an expiration period. And fill out personal information.

Now you can list all your keys:

gpg2 --list-secret-keys

Copy a keyid of your key and distribute it as follows:

gpg2 --keyserver hkp://pool.sks-keyservers.net --send-keys [keyid]

A bit more details on this step are here.

Gradle script

Now we can touch the project. Here are changes which should be made to your build.gradle:

...
apply plugin: 'maven'
apply plugin: 'signing'

task sourcesJar(type: Jar) {
    classifier = 'sources'
    from sourceSets.main.allSource
}

task javadocJar(type: Jar) {
    classifier = 'javadoc'
    from scaladoc
}

artifacts {
    archives javadocJar, sourcesJar
}

signing {
    sign configurations.archives
}

uploadArchives {
    repositories {
        mavenDeployer {
            beforeDeployment { MavenDeployment deployment -> signing.signPom(deployment) }

            repository(url: "https://oss.sonatype.org/service/local/staging/deploy/maven2/") {
                authentication(userName: ossrhUsername, password: ossrhPassword)
            }

            snapshotRepository(url: "https://oss.sonatype.org/content/repositories/snapshots/") {
                authentication(userName: ossrhUsername, password: ossrhPassword)
            }

            pom.project {
                name 'testcontainers-scala'
                packaging 'jar'
                description 'Scala wrapper for testcontainers-java'
                url 'https://github.com/dimafeng/testcontainers-scala'

                scm {
                    connection 'scm:git:git@github.com:dimafeng/testcontainers-scala.git'
                    developerConnection 'scm:git:git@github.com:dimafeng/testcontainers-scala.git'
                    url 'git@github.com:dimafeng/testcontainers-scala.git'
                }

                licenses {
                    license {
                        name 'The MIT License (MIT)'
                        url 'https://opensource.org/licenses/MIT'
                    }
                }

                developers {
                    developer {
                        name 'Your name'
                        email 'your email'
                    }
                }
            }
        }
    }
}

This code allows publishing of releases and snapshots to oss.sonatype.org. You just need to adjust pom.project section to match your project information.

And to make it all work, we need to modify ~/.gradle/gradle.properties by adding a several properties:

signing.keyId=#keyId
signing.password=#yourPassword
signing.secretKeyRingFile=/Users/user/.gnupg/secring.gpg
ossrhUsername=#Sonatype's JIRA username
ossrhPassword=# password

More information on it is here.

Publishing

It seems it’s time to deploy your artifact. Make sure your work with non-snapshot version and then execute:

./gradlew uploadArchives

Once it’s done, you need to promote your release. To do so, you have to go to this page. In the table with repositories, you’ll see one that looks like your group id but without dots. Go in there and then click ‘Close’ in the top menu. Refresh the view in a few seconds and go to the ‘Activity’ tab - if you did everything correctly on the previous steps, you’ll see no errors and a button ‘Release’ will be enabled (it’s located to the right of ‘Close’ button), click on it and that’s pretty much it.

Enjoy having your own artifact in maven cental!

Not obvious scala features for java developers

Mon, 30 May 2016 00:00:00 +0000

A week ago I finished reading of Scala for the Impatient by Cay Horstmann. Here are some lessons which I learned from this book.

1. Method parentheses

For warming up, here’s the first tip - it’s well known that you can omit using of parentheses for methods without parameters. But there is a rule in scala world: you can omit parentheses on methods which don’t mutate a state.

class State {
  private var value = 0

  def change(): Unit = {
    value = Random.nextInt()
  }

  def current = value
}

...

s.change() // With parentheses
s.current // Without parentheses

2. Traits vs abstract classes

Java 8 brings a new experience to interface usage - interface methods can have a default implementation. But scala eliminates the distinction even more (moreover, it works on JVM < 8): scala traits can have a default method implementation, abstract and defined values and variables. That’s why, in scala world, there is a rule: if you don’t know that to use traits or abstract classes - use traits, you’ll always be able to change them to abstract classes.

trait Car {
  val color: String

  def print() = {
    println(s"Car is ${color}")
  }
}
...
new Car {
  override val color: String = "red"
}.print()

Why is it good

You can use traits as mixins. You even can use it during object creation:

abstract class Car {
  def specs() = {
    print(s"Car with ${wheels} wheels")
  }

  def wheels: Int
}

trait FourWheel {
  def wheels = 4
}
...
(new Car with FourWheel).specs() // Output: Car with 4 wheels

Why is it bad

You can’t define constructor
Most features aren’t supported in java. So, you loose java interoperability
This approach has lesser performance

3. ProcessBuilder DSL

There is a nice feature ti execute external one or more external processes from scala code using very convinient DSL:

object CMDExample extends App {
  println("ls" !) //(1)
  println("ls" !!) //(2)
  println("ls" #| "grep gradle " !!) //(3)
}

Executes ls within working directory and prints to stdout, then returns exit code.
The save as 1. but returns command output as a string
Equivalent of ls | grep gradle

4. Regex

Scala provides an easy way to work with java RegEx api. There’s a class and companion object which hold all this convenience - scala.util.matching.Regex.

To create an instance of Regex, you can use method r mixed to a string. It may looks like this: "\\d+".r. Then, you can work with it as follows:

Find matches

val regex = s"""<a\\s+href=\"([^\"]*?)\"\\s*>([^<]*?)</a>""".r
val regex(href, name) = "<a href=\"test.html\">test</a>"
println(href + ":" + name) // Output: test.html:test

For multiple matchings:

val text =
    s"""
       |<a href="index.html">index</a>
       |<a href="secondary.html">secondary</a>
   """.stripMargin

for(regex(href,_) <- regex.findAllIn(text)) {
    println(href)
}

Replace

println(regex.replaceAllIn(text, { m => m.group(2) }))

The output of this line will be a string where all a tags replaced with a content of those tags.

5. Structural type

Structural type is very interesting feature. It allows to define method argument of any type but with requirement of presence of specific methods. This is how it looks like:

def printInfo(obj: {def info: String}) = println(obj.info)

class A {
  def info: String = "A"
}

class B {
  def info: String = "B"
}

printInfo(new A)
printInfo(new B)

Netty Hello World

Sun, 08 May 2016 00:00:00 +0000

When you read an article about async programming in java you will definitely find at least mentions of Netty. I’ve heard about this library, I know that many high-performance frameworks based on Netty (Akka, Vert.x, the full list is here), but I’ve never tried it. So it’s time to write a hello world application using Netty and scala.

What is Netty

Netty is an asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients.

Simply put, Netty is going to be useful if you want to write a server that handles all requests in one thread (technically not one) instead of using a separate thread for each client request. What’s wrong with a thread per request approach? - when we get a high load, we need to have a lot of threads to serve all clients, but having many threads causes high memory consumption and many thread context switches (the second reason is more important due to the fact that it is an expensive operation). This approach is very popular, for example, in javascript world. Node.js http server works exactly in this way.

Netty operates with Channels which are abstractions over a socket. As you may see, data can be read from or written to them - all these operations will be non-blocking. In addition, you can combine channels in groups and execute the same operations as for individual channels. If you’re creating server application you need some of the implementations of ServerSocketChannel. To start working with a server channel, we have to introduce another important thing - thread pools which all the magic will be run on. There are two thread pools which should be passed to the server channel: boss and workers. The boss one is going to accept incoming connections while the workers process these accepted connections.

What are we going to create

As a demo application, I’m going to code simple app that will be capturing image from webcam and transfer it over websocket to the client side, then javascript will draw it on the canvas - it’s cross-browser and cross-platform solution, you can watch a video even on a mobile device (without audio though).

To capture images from the camera, we’ll use webcam-capture. It’s written in pure java and has very simple api.

Server Side

All the code split into two parts: server side and client side. Due to the fact that I’m learning scala, the server side will be written in scala in java-style (I’m trying to adopt canonical, functional style, but this is tough to do very fast)

import java.awt.Dimension
import java.io.ByteArrayOutputStream
import java.util.concurrent.Executors
import javax.imageio.ImageIO

import com.github.sarxos.webcam.Webcam
import io.netty.bootstrap.ServerBootstrap
import io.netty.buffer.Unpooled
import io.netty.channel.group.{ChannelGroup, DefaultChannelGroup}
import io.netty.channel.nio.NioEventLoopGroup
import io.netty.channel.socket.SocketChannel
import io.netty.channel.socket.nio.NioServerSocketChannel
import io.netty.channel.{ChannelHandlerContext, ChannelInitializer, SimpleChannelInboundHandler}
import io.netty.handler.codec.http.websocketx.{BinaryWebSocketFrame, WebSocketFrame, WebSocketServerProtocolHandler}
import io.netty.handler.codec.http.{HttpObjectAggregator, HttpServerCodec}
import io.netty.handler.logging.{LogLevel, LoggingHandler}
import io.netty.util.concurrent.GlobalEventExecutor

import scala.concurrent.{ExecutionContext, Future}

object Application {

  implicit val ec = ExecutionContext.fromExecutor(Executors.newSingleThreadExecutor())

  val WEBSOCKET_PATH = "/websocket"
  val FPS = 30

  def main(args: Array[String]) {

    val allChannels = new DefaultChannelGroup("all", GlobalEventExecutor.INSTANCE)

    startCapture(allChannels)
    startServer(allChannels)
  }

  def startServer(group: ChannelGroup) = {
    val bossGroup = new NioEventLoopGroup(1)
    val workerGroup = new NioEventLoopGroup
    try {
      new ServerBootstrap()
        .group(bossGroup, workerGroup)
        .channel(classOf[NioServerSocketChannel])
        .handler(new LoggingHandler(LogLevel.INFO))
        .childHandler(new ChannelInitializer[SocketChannel] {
          override def initChannel(ch: SocketChannel): Unit = {
            ch.pipeline
              .addLast(new HttpServerCodec)
              .addLast(new HttpObjectAggregator(65536))
              .addLast(new WebSocketServerProtocolHandler(WEBSOCKET_PATH, null, true))
              .addLast(new WebSocketFrameHandler(group))
          }
        }).bind(8080).sync
        .channel
        .closeFuture.sync
    } finally {
      bossGroup.shutdownGracefully()
      workerGroup.shutdownGracefully()
    }
  }

  def startCapture(group: ChannelGroup) = {
    val webcam: Webcam = Webcam.getDefault
    webcam.setViewSize(new Dimension(640, 480))
    webcam.open

    Future {
      while (true) {
        val outputStream = new ByteArrayOutputStream
        ImageIO.write(webcam.getImage, "jpg", outputStream)
        group.writeAndFlush(new BinaryWebSocketFrame(Unpooled.copiedBuffer(outputStream.toByteArray)))
        Thread.sleep(1000 / FPS)
      }
    }
  }
}

class WebSocketFrameHandler(group: ChannelGroup) extends SimpleChannelInboundHandler[WebSocketFrame] {

  override def userEventTriggered(ctx: ChannelHandlerContext, evt: scala.Any): Unit = {
    if (evt == WebSocketServerProtocolHandler.ServerHandshakeStateEvent.HANDSHAKE_COMPLETE) {
      group.add(ctx.channel())
    } else {
      super.userEventTriggered(ctx, evt)
    }
  }

  protected def channelRead0(ctx: ChannelHandlerContext, frame: WebSocketFrame) {
    // do nothing
  }
}

As you see, we don’t have many lines of code (thanks, scala!). Let’s go through the all key parts of this code. There are two methods: the first one connects to default camera and starts capturing images (the details of this process won’t be covered in this article), the second one starts a web server. To define a server, there is a special class ServerBootstrap which is a ServerChannel but with a simplified interface for server creation. It provides builder-style approach for its configuration. Here are main configuration steps:

group - takes two event loops, the firs one is for accepting incoming connections, the second one is for data processing
channel - takes a type of the channel that will be created
childHandler - takes a ChannelHandler that will initialize a channel. In our case, it is a ChannelInitializer. Its implementation is described below

The last step is bind, that binds our channel to given port and returns a ChannelFuture. That ChannelFuture is used for correct closing of the server channel.

Now let’s go back to ChannelInitializer. The most interesting part of it is pipeline. Pipeline passes all the date through a chain of ChannelHandlers. In our case, we have HttpServerCodec which is a http protocol implementation, WebSocketServerProtocolHandler easily implements WebSockets (it requires HttpObjectAggregator) and the last one WebSocketFrameHandler which is our handler that does simple thing - it add newly connected client to the group of channels. This group is used in parallel thread that captures image and puts the binary into the group. Very simple!

Client side

As per our requirements, the client application should connect to a websocket and render an image once it comes to the client side. Here is all that we need:

<canvas id="viewport"></canvas>
<script type="text/javascript">
    var canvas = document.getElementById('viewport');
    var context = canvas.getContext('2d');

    var socket;
    if (window.WebSocket) {
        socket = new WebSocket("ws://localhost:8080/websocket");
        socket.binaryType = "arraybuffer";
        socket.onmessage = function (event) {
            draw(event.data);
        };
    } else {
        alert("Your browser does not support Web Socket.");
    }

    function draw(imgData) {
        var b64imgData = arrayBufferToBase64(imgData);
        var img = new Image();
        img.src = "data:image/jpeg;base64," + b64imgData;
        canvas.width = img.width;
        canvas.height = img.height;
        context.drawImage(img, 0, 0, 640, 480);
    }

    function arrayBufferToBase64(buffer) {
        var binary = '';
        var bytes = new Uint8Array(buffer);
        var len = bytes.byteLength;
        for (var i = 0; i < len; i++) {
            binary += String.fromCharCode(bytes[i]);
        }
        return window.btoa(binary);
    }
</script>

This is pure javascipt. We even don’t need jQuery. The first part is standard WebSocket initialization - nothing to comment on it. There are two functions which should be described. arrayBufferToBase64 takes ArrayBuffer (WebSocket in socket.binaryType = "arraybuffer" mode uses this type of object to pass binaries) and converts to Base64 string. I haven’t written this function, I’ve stolen it from here. And the last key function here is draw - when we have Base64 binaries of an image we can draw it on a canvas and this is a typical code to do so.

dimafeng.com

neotypes: Akka-http + neo4j

Akka-http + neo4j

Environment configuration

Data access layer

Akka-http

Wrap it up

Conclusion

Flexible alerting for ELK

Disclaimer

Note

Web playgroud

Real configuration

Rule execution metadata

Running the rule in background

What’s next?

Pragmatic view of the modern front-end application development

Disclaimer

Note

What a modern web application is

Project structure

Dependency management

Note

TypeScript

Webpack

Linting

Build steps

Project’s stack

React.js

Communication with a backend API

MobX

Note

Dependency Injection

Testing

Dev Tools

Wrapping up

5 drawbacks to liking SBT

1. Speed

2. Project properties

3. Test exclusion

4. Publishing to maven central

5. Releasing

Conclusion

Distributed systems cheat sheet

Disclaimer

What a distributed system is

Physical factors

Time and Order

CAP

Consistency

Replication

Synchronicity

Synchronous

Asynchronous

Semi-synchronous

Node relationship

Master/Slave (single copy)

Multi-master

Consensus

Two-Phase commit

Three-Phase commit

Paxos

RAFT

Partitioning

Horizontal partitioning

Links

How to manage your dotfiles

What are dotfiles and (semi-)standard way

Which dotfiles to work with

install.sh

.bash_profile

.zshrc

.aliases

.functions

.gitconfig

.osx

.vimrc

Integration testing using docker and scala

Testcontainers

Assumptions