Volkan Yazıcı

The State of Lightweight Threads for the JVM

2019-12-17T21:57:00Z

This week I took the stage at bol.com with a presentation titled The State of Lightweight Threads for the JVM. The abstract of the presentation was as follows:

Thanks to your constantly crashing desktop apps, you know about processes. Microservice flavored web development taught us about threads too. Then came callbacks, shortening lifetime of millions of developers. Do you recall the day you shed tears when Java shipped a Future that you can complete? I know you haven’t forgotten how Reactive Streams occupied our agendas with RxJava,Reactor, and infinite stack traces. Now newspapers are yelling about coroutines, fibers, lightweight threads! We poor souls… In this presentation, I aimto give a 2,000ft overview of where do we come from, where are we heading to, and what does all this mean for the Java Virtual Machine, aka, JVM. Are you a Java developer? Thread circus owner? Reactive ninja? Kotlin fan? COBOL enthusiast? Algol pensioner? There is something I will tell you. (Spoiler: I will as well talk about Project Loom, structured concurrency, and scoped variables.)

The slides are available in PDF and ODP formats.

Elasticsearch Survival Guide for Developers

2019-04-25T11:44:00Z

For some months, I have been jotting down notes on best practices that I wish I would have known when I first started developing applications running against Elasticsearch. Even though the following collection tries to communicate certain ideas in Java, I believe almost each of such cases apply to every other programming language with almost no or minor changes. I tried to avoid repeating content that has already been covered in tutorials and the Elasticsearch documentation. The listed principles are all derived from my personal point of view, I strived to share only the ones that I can justify with either facts or experience.

Before proceeding further, I would like to thank David Turner and William Leese for their valuable feedback. A Chinese translation of this article is kindly featured on CSDN by Ming Yi Tian Xia.

Mapping
Setting
Querying
Strategy

Mapping

Here I share Elasticsearch mapping related tips.

Avoid `nested` fields

Under the hood, each Elasticsearch document corresponds to a Lucene document, most of the time. This promise is broken for fields of type nested. There each field is stored as a separate document next to the parent Lucene one. The impact of this being:
- Querying on nested fields is slower compared to fields in parent document
- Retrieval of matching nested fields adds an additional slowdown
- Once you update any field of a document containing nested fields, independent of whether you updated a nested field or not, all the underlying Lucene documents (parent and all its nested children) need to be marked as deleted and rewritten. In addition to slowing down your updates, such an operation also creates garbage to be cleaned up by segment merging later on.

In certain ocassions, you can flatten nested fields. For instance, given the following document:

{
    "attributes": [
        {"key": "color", "val": "green"},
        {"key": "color", "val": "blue"},
        {"key": "size", "val": "medium"}
    ]
}

You can flatten it as follows:

{
    "attributes": {
        "color": ["green", "blue"],
        "size": "medium"
    }
}

Have a strict mapping

Do you know how many times I witnessed a production failure due to a new field first receiving a long value where the rest of the values are of type double? After the first received long, Elasticsearch creates the field, sets its type to long, and shares this mapping change with the rest of the nodes in the cluster. Then the rest of the double values are simply rejected due to type mismatch.

Have a strict mapping to avoid surprises.
Don’t blacklist, just whitelist.
Avoid using dynamic templates – they are just gateway drugs.
Disable date detection, which is on by default.

Don’t analyze fields of type `string` unless necessary

By default, a freshly inserted string field is assigned of type text, which incurs an analysis cost. Unless you need fuzzy matching, and just want filtering, use type keyword instead. This is small amendment of the strict mapping bullet point.

Settings

Here I share Elasticsearch cluster settings related tips.

Avoid oversharding

One of the greatest strengths of Elasticsearch is sharding, that is, splitting the data into multiple nodes to exploit parallelization. There are many myths surrounding this subject. Recall that sharding of an index cannot be changed once it is set. This makes oversharding a very common pitfall for newcomers. Make sure you have done your homework (that is, RTFM, such as this one) before taking any decisions.

Unlearn every hack for tuning merges

Elasticsearch is in essence yet another distributed Lucene offering, just like Solr. Under the hood, every Elasticsearch document corresponds to a Lucene document, most of the time. (There are certain exceptions to this rule like nested fields, though this generalization is pretty accurate.) In Lucene, documents are stored in segments. Elasticsearch in the background continuously maintains these Lucene segments by means of the following two patterns:

In Lucene, when you delete and/or update a document, the old one gets marked as removed and a new one gets created. Elasticsearch keeps track of these dead documents and compacts such segments that are highly polluted by rebuilding them.
Newly added documents might yield to segments of imbalanced sizes. Elasticsearch might decide to merge these into bigger ones for optimization purposes.

This aforementioned compaction is referred to as segment merges in Elasticsearch terminology. As you can guess, merges are highly disk I/O- and CPU-bound operations. As a user, you would not want to have them ruining your Elasticsearch query performance. As a matter of fact, you can avoid them completely in certain circumstances: Build the index once and don’t change it anymore. Though this condition might be difficult to meet in many application scenarios. Once you start to insert new documents or update existing ones, segment merges become an inevitable part of your life.

An on-going segment merge can significantly damage the overal query performance of the cluster. Do a random search on google and you will find many people looking for help to reduce the performance impact and many others sharing certain settings that worked for them. Over the last years, there were two particular patterns I observed in these cries for help: they existed since the earliest of Elasticsearch versions (so everybody agrees that it used to hurt and is still hurting) and the majority of the mentioned settings have become deprecated over time (or even worse, removed completely). So my rules of thumb for tuning merges is as follows:

Unlearn every hack you heard about tuning merges. It is an operation tightly coupled with the internals of Elasticsearch and subject to change without providing a backward compatibility. There is no secret knob to make it run faster; it is like the garbage collector in JVM, VACUUM in PostgreSQL, etc.
Find the sweet spot for the translog flushes. Try relaxing index.translog.sync_interval and index.translog.flush_threshold_size settings until you stop seeing benefits for your usage pattern.
Adapt index.refresh_interval to your needs. Imagine you first bootstrap an index and later on occasionally perform small updates on it. In such a case, start with a loose (even disabled!) refresh_interval and make it tighter after bootstrap.

Note that in recent versions, if you are indexing but not searching then no refreshes will take place at all. Quoting from Dynamic Index Settings:

If this [index.refresh_interval] setting is not explicitly set, shards that haven’t seen search traffic for at least index.search.idle.after seconds will not receive background refreshes until they receive a search request.

Pay attention to JVM memory settings

Elasticsearch can yield dramatic performance characteristics depending on two primary memory settings: JVM heap space and the amount of memory left to the kernel page cache. I am not going to dive into these details here, because they are pretty well documented. This is your reminder to not blindly set the Elasticsearch JVM heap size judging from your past non-Elasticsearch JVM application experiences.

Querying

Below I collected tips that you can take advantage of (or better stick to) while querying Elasticsearch.

Compare-and-swap over `_version` field is poor man’s transactions

I believe you have already figured out that Elasticsearch doesn’t support transactions. Though you can leverage the _version field in a CAS-loop to provide integrity at least on a single document basis. An example demonstration of this trick can be summarized as follows:

String id = "123";
for (;;) {
    EsDocument prevDoc = store.findById(id);
    long prevVersion = prevDoc.getVersion();
    Object prevSource = prevDoc.getSource();
    Object nextSource = update(prevSource);
    boolean updated = store.updateByIdAndVersion(id, prevVersion, nextSource);
    if (updated) {
        break;
    }
}

Clearly this trick doesn’t stretch to multiple indices or parent/child relations.

Warning: Usage of _version within CAS-loops are deprecated due to known issues. In Elasticsearch version 6.6 and onwards, one should rather use _seq_no and _primary_term fields instead. Though the gist of this practice is still the same. See Optimistic Concurrency Control for details.

Try splitting complex queries

If you have complex queries with both, say, filter and aggregation components, splitting these into multiple queries and executing them in parallel speeds up the querying performance in most cases. That is, in the first query just get the hits using the filter then in the second query, just get the aggregations without retrieving hits, that is, size: 0.

Know your numeric types

Many JSON parsers reach for various optimizations to provide efficient read/write performance. For instance, Jackson, the de facto JSON parser in the Java world, picks the primitive with the smallest memory footprint that can store the number passed by JSON. Hence after reading a field value via Jackson, you might end up getting an int, long, double, etc. Once you get a, say, double, it is highly likely that you had already lost precision and/or will be losing precision while serializing it back to JSON again. To avoid such surprises, prefer BigDecimal over float or double. Using BigInteger for integral numbers is a safe bet too. (See USE_BIG_DECIMAL_FOR_FLOATS and USE_BIG_INTEGER_FOR_INTS Jackson configurations for details.)

Don’t use Elasticsearch Transport/Node client in your Java application (and always use JSON over HTTP)

Elasticsearch is written in Java and its query model classes implement custom (de)serialization methods using Jackson’s JsonGenerator and JsonParser. This way, thanks to Jackson, a model instance can be (de)serialized to both JSON (text) and SMILE (binary) formats without breaking a sweat. Logically, Elasticsearch uses the binary format for communication within the cluster due to performance reasons. Using JsonParser for parsing SMILE has a slight caveat: A schema cannot always be evolved in such a way that backwards compatibility is guaranteed. Indeed this is not a problem for an Elasticsearch cluster; all the nodes (hopefully!) run the same version. Though using SMILE in your application means that you might need to shutdown your application, upgrade it to a newer version which is using the models of the new Elasticsearch you are about to deploy in parallel.

What about performance between JSON and SMILE? Even Elastic’s own data intensive products such as Logstash and Kibana have replaced SMILE with JSON. It is highly unlikely that your bottleneck while querying Elasticsearch would be serialization. Further Jackson is an excelling library in JSON serialization efficiency. Hence, to be on the safe side, just stick to JSON over HTTP.

Use the official Elasticsearch REST client in your Java application

The official driver will be maintained through the lifetime of the project. It will implement community approved features from its competitors. Unless something goes terribly wrong, the official client will either assimilate the others by simply including their features or the unofficial clients will just rust away due to not being able to keep up with the rate of new features being introduced in Elasticsearch and finally disappear. That being said, the official REST client is a piece of crap for two main reasons:

It has a leaky abstraction over Apache HTTP Client whose configurations you need to deal with while tuning the client.
Do you recall the query model classes I mentioned above? I have some bad news for you: Model classes are entangled with the server code and the REST client uses those classes. So what does this mean for you? Well… Adding the REST client in your dependencies will drag the entire Elasticsearch milkyway into your JAR Hell.

Eclipse Vert.x works around this entanglement in a yet another entangled way. Though I doubt if it will be granted a long life given the reasons I listed above.

In summary, the official REST client (unfortunately) is still your best bet.

Don’t use Elasticsearch query models to generate cache keys

Or more specifically:

[If you want your hash keys to be consistent between different JVM processes, JVM upgrades, and JVM restarts,] don’t use Object#hashCode().
The JSON generated by Elasticsearch query models for semantically identical queries are not necessarily identical. It took more than a year to figure this out ourselves. Hence, don’t take the query model generated JSON as your source of hash. (My guess is that somewhere in the JsonGenerator-based serializers, they are probably iterating over a java.util.Map or java.util.Set whose order for identical content varies under different conditions.)

Ok. So what else is left? How should we do it?

You query Elasticsearch due to a request you have just received, right? Does that request has its own application level model? Good. There you go. Use that one as a hash source.
Your application level request model is too complex to generate a proper hash key? Doh! Ok. Don’t tell anyone that I told you this: Parse Elasticsearch generated JSON using Jackson and let Jackson serialize it one more time back to JSON, but this time instruct Jackson to sort the keys.

Don’t use HTTP caching for caching Elasticsearch responses

Many people fall in the trap of fronting their Elasticsearch cluster with an HTTP cache such as Varnish due to its convenience and low barrier of entry. This seductive approach has certain shortcomings:

When using Elasticsearch in production, it is highly likely you will end up having multiple clusters due to various reasons: resiliency, experimentation room, zero downtime upgrades, etc. Then,
- once you front each cluster with a dedicated HTTP cache, 99% of the cache content will just be duplicated.
- if you decide to use a single HTTP cache for all clusters, it is really difficult to programmatically configure an HTTP cache to adopt the needs of the ever changing cluster states. How will you communicate the cluster load to let the cache balance the traffic. How will you configure scheduled or manual downtimes? How will you make the cache gradually migrate the traffic from one to another during maintanence windows?
As mentioned above, HTTP caches are difficult to command programmatically. When you need to manually evict one or more entries, it is not always as easy as a DELETE FROM cache WHERE keys IN (...) query. And let me warn you, you are gonna need that manual eviction sooner or later.

Use sliced scrolls sorted on `_doc`

Scrolls are the vehicle that Elasticsearch provides allowing you to scan its entire dataset for large reads. They are functionally (and surprisingly, internally too) pretty much similar to RDBMS cursors. Though most people don’t get them right in their first attempt. Here are some basics:

If you reach for scrolls, you are probably reading quite some data. It is highly likely slicing will help you improve the read performance significantly.
The order doesn’t matter in your reads? This is your lucky day! Just sort on _doc field and there’s a +20% read speed up without any other changes. (_doc is a pseudo field to let Elasticsearch use the order documents are in on disk.)
The scrollId might (and does!) change between calls. Hence make sure you are always scrolling using the most recently retrieved scrollId.
Reaching for REINDEX? Did you enable slicing there too? Good.

Prefer `GET /index/type/{id}` over `POST /index/_search` for single document retrieval

Elasticsearch uses different thread pools to handle GET /index/type/{id} and POST /index/_search queries. Using POST /index/_search with payload {query: {"match": {"_id": "123"}}} (or something similar) occupies your search-dedicated thread pool. Under heavy load, this will worsen your both search and single document fetch performance. Simply just stick to GET /index/type/{id}.

Use `size: 0` and `includes`/`excludes` wisely

This tip I guess applies to any storage engine of your preference: Avoid retrieving (or even storing!) content unless necessary. I have witnessed Elasticsearch showing dramatic performance differences before and after adding a size: 0 clause.

Implement proper backpressure while querying

Yet another database-independent tip: There is no point in hammering your database to the point of making it choke. Consider the following Java snippet employing an imaginary database client:

void findUserById(String id, Callable<User> callback) {
    dbClient.find("user")
            .filter("id", id)
            .first(callback::apply);
}

Assuming dbClient is implemented in a non-blocking and asynchronous fashion – that is, each request is enqueued to be delivered and each response handler is enqueued to react on incoming payloads – what would happen if your database can handle at most 1 req/sec while your application perfectly receives, handles, and passes on 10 req/sec? Let me draw you a shallow probability tree depicting consequences of such an incident.

Your database gets more load than it can take. If your database has proper backpressure mechanics, which most don’t possess, including Elasticsearch,

Then it will start choking and eventually throw up. This will get reflected as query errors on the application side. If your application is equipped with backpressure mechanics as well, it can kindly reflect this back to the caller.
Otherwise,
1. even the simplest database queries will start suffering due to heavy load.
2. the database process queue will overgrow.
  1. Excessive growth of the queue (that is, no backpressure mechanics in place) will start stressing the process memory.
  2. The requests that succeed in making from the queue to an executor thread will highly likely already become deprecated. That is, the database will be doing work that is of no use to anybody.
3. above two points drawn from the process queue overgrow of the database, also apply to the application.

Unfortunately there is no silver bullet or a step-by-step guide to implementing backpressure for a particular application. This in a way makes sense, each has its own domain-specific requirements. That being said, I can share my personal best practices:

Use performance benchmarks of your application (You have performance benchmarks, right?) to estimate an upper bound on the load that your application still delivers acceptable performance. Enforce this limit in your application via a rate limiter. Please, please, please don’t block the carrier thread using the rate limiter! Rather just communicate the backpressure to your client, for instance, by means of an HTTP 503 (SERVICE UNAVAILABLE) status code.
Avoid using thread pools with an unbounded task queue. Java programmers, all your RxJava/Reactor schedulers are by default backed by a ScheduledThreadPoolExecutor (the one and only ScheduledExecutorService implementation in the Java standard library) and that internally uses an unbounded task queue, unfortunately. See this Stack Overflow post and this concurrency-interest discussion on how to work around that.
If your application is a pipe between two resources (e.g., from a Pubsub queue to an Elasticsearch cluster) make sure your producers react to consumers’ backpressure. That is, if the consumer latency starts increasing, you better start slowing down the producer.

Provide explicit timeouts in queries

Almost all Elasticsearch API endpoints allow the user to specify a timeout - use it. This will help both your application and your Elasticsearch cluster: Spot and shrug off unexpected long running operations, save associated resources, establish a stable SLA with no surprises, etc.

Don’t block the Elasticsearch client I/O threads (and know your threads)

This tip is also database independent: Don’t block I/O threads of an external resource for another external resource. Let me demonstrate this pitfall with a snippet:

public void enrichUserById(String id, Function<EsDocument, EsDocument> enricher, Runnable callback) {
    esClient.findUserById(id, user -> {
        EsDocument enrichedUser = enricher.apply(user);
        esClient.saveUserById(id, enrichedUser, callback);
    });
}

What is mostly unintentional here is that: Both enricher.apply(user) and callback.run() will get executed in the Elasticsearch client I/O thread. Here I see two common cases:

If both functions don’t incur any other I/O calls (except the ones that are again reaching for Elasticsearch) and this is the only place in the entire application where you access to Elasticsearch, then this is a good practice. You repurpose Elasticsearch client I/O thread for a CPU-intensive post-processing. Almost no thread context-switch costs. On a 4 core machine, with 4 threads dedicated to Elasticsearch client I/O, you will get an almost optimal performance given your usage pattern.

If both or any of the functions internally perform other I/O calls and/or there are multiple places in the application where Elasticsearch client is used for different purposes, then you are occupying the Elasticsearch client I/O threads for something unrelated whereas these threads could have just been serving yet another Elasticsearch request. In such cases, it is better to employ task-specific thread pools and avoid exhausting Elasticsearch client I/O loop unnecessarily:

public void enrichUserById(String id, Function<EsDocument, EsDocument> enricher, Runnable callback) {
    esClient.findUserById(id, user -> {
        computationExecutor.submit(() -> {
          EsDocument enrichedUser = enricher.apply(user);
          esClient.saveUserById(id, enrichedUser, () -> {
              computationExecutor.submit(callback);
          });
        });
    });
}

Don’t write Elasticsearch queries with JSON templates injecting variables

Don’t ever do this:

{
    "query": {
        "bool": {
            "filter": [
                {
                    "term": {
                        "username": {
                            "value": {{username}}
                        }
                    }
                },
                {
                    "term": {
                        "password": {
                            "password": {{password}}
                        }
                    }
                },
            ]
        }
    }
}

You just fell into a half-century old trap: SQL injection, adopted for Elasticsearch. No matter how smart your character whitelisting, escaping routines are, it is just a matter of time until someone passes a malicious username and/or password input that would expose your entire dataset. I also personally find it a pretty bad practice to render a curly brace ({, }) rich structured text via a templating language (e.g., Moustache, Handlebars) which uses the very same curly braces for its own directives.

There are two safe approaches that I can recommend to generate dynamic queries:

Use the query models provided by the Elasticsearch (official) client. (This works for Java pretty well.)
Use a JSON library (e.g., Jackson) to build the JSON tree and serialize it to JSON.

Prefer your own JSON serializer over the one provided by Elasticsearch clients

Many Elasticsearch clients allow you to pass a generic JSON object and serialize it to JSON before passing it over the wire. For instance, the official Elasticsearch REST client for Java allows java.util.Map<String, Object> instances as source. Then it uses its own JSON serializer to translate these models into JSON. While this works fine for vanilla Java, which most of the time is sufficient to get the message across in tutorials, most real world applications have more complex class structures that necessitate custom serialization. For instance, speaking of Java client, how does it serialize Guava models? What about the new date and time classes introduced in Java 8? What will happen to all your @JsonProperty-, @JsonSerializes-, etc. annotated classes? Hence it is always a better practice to employ your own serialization and pass a byte[] as source to the client. That will save you from any nasty surprises.

Strategy

In this last section I collected convenient strategic (whatever that means) practices which address concerns not covered above.

Always (try to) stick to the latest JVM and Elasticsearch versions

Elasticsearch is a Java application. (Surprise, surprise!) Like every other Java application it has its hot paths and garbage collection woes. Almost every new JVM release will bring you more optimizations that you can take advantage of without breaking a sweat. Note that due to the low-level performance hacks exploited in Lucene, which is used internally by Elasticsearch, Lucene is sort of fragile to JVM upgrades, particularly involving garbage collector changes. Fortunately, Elasticsearch has an official page listing supported JVM releases and garbage collectors. Always check these pages out before attempting any JVM upgrades.

Elasticsearch upgrades are also a source of free performance gains. I have never experienced a regression after Elasticsearch upgrades. That said, your milage may vary and this is why you should have proper integration tests in place.

Use Elasticsearch complete and partial snapshots for backups

Elasticsearch lets you easily take snapshots of an index either completely or partially against an existing snapshot. Depending on your update patterns and index size, find the best combination for your use case. That is, for instance, 1 complete snapshot at 00:00 and 3 partial snapshots at 06:00, 12:00, and 18:00. It is also known to be a good practice to have them stored at AWS S3 or GCP Storage. There are plugins to facilitate these scenarios.

As in every backup solution, make sure you can restore them and practice this a couple of times. In case of a post-failure restoration, you might need to engineer your own replay mechanisms to reach the last stable state just before the crash. Leveraging queues supporting custom retention periods (e.g., keep all the messages received in the last 2 days) for this purpose is a practice employed a lot.

Have a continuous performance test bed

Like any other database, Elasticsearch shows varying performance under different conditions: index, document sizes; update, query patterns; index, cluster settings; hardware, OS, JVM versions, etc. It is difficult to keep track of each knob to observe its impact on overall performance. Make sure you have (at least) daily performance tests to help you narrow down a recently introduced change contributing to lower performance.

This utopic test bed is easier said than done. You will need to make sure that the test environment has representative data of production, (preferably) identical configuration to production, complete coverage of use cases, and provides a clean slate (including the OS cache!) for each test to avoid retaining the effects of a previous run. I know, quite a list. But it pays off.

Use aliases

Now I am gonna tell you something quite opinionated, though backed by experience: Never query against indices, but rather against aliases. Aliases are pointers to actual indices. You can group one or more indices under a single alias. Many Elasticsearch indices have an internal context attached to the index name, such as, events-20190515. Now you have two choices in the application code while querying against events-* indices:

Determine the index name on the fly via a certain date pattern: events-YYYYMMDD. This approach has two major drawbacks:
- The need to fallback to an index of a particular date necessitates the entire code base to be engineered accordingly to support such an operation.
- Putting all clock synchronization issues aside, at midnight, you need to make sure that the next index is there.
Create an events alias pointing to the events-* index you want the application to use. The component responsible for the creation of the new indices can atomically switch the alias to the new index. This approach will bring two notable benefits:
- It doesn’t suffer from the drawbacks of the previous approach.
- The application code is way more simpler by just pointing to the events index everywhere.

Avoid having big synonym collections

Elasticsearch supports both index- and query-time synonyms. These are powerful shotguns with a good track record of shooting its wielder in the foot. No search engine is complete without synonyms, hence they have pretty valid use cases. That said, the following implications need to be kept in mind while employing them:

Index-time synonyms increase the index size and create extra runtime overhead.
Query-time synonyms doesn’t add to the index size, but as their name implies, creates extra runtime overhead.
Using synonyms, it is pretty easy to unintentionally break something while trying to fix some other thing.

Continuosly monitor the impact of synonyms on the performance and try to write tests for each synonym added.

Force merge and increase operation bandwidth before enabling replicas

A really common Elasticsearch use case is to periodically (once every couple of hours) create an index. There is a really good SoundCloud article on how to achieve the optimal performance. Quoting from that compilation, there are the following items that I particularly find a “must”.

Always enable replicas after completing the indexing.
Before enabling replicas, make sure you
- shrinked the index size by forcing a merge and
- temporarily increased the replica transmission bandwidth until allocation has finished, that is, indices.recovery.max_bytes_per_sec.

Record application-level metrics

Kibana provides quite solid insights into Elasticsearch performance: indexing, search latency and throughput, flush, merge operations, etc. Once you enhance this view with extra JVM (GC pauses, heap size, etc.) and OS (CPU usage, disk I/O, kernel caches, etc.) metrics, you will get a rock solid monitoring dashboard. That said, this is not enough. If used by a single application or more, Elasticsearch will get hit by various access patterns. Imagine a part of your software trying to shovel off 10 million documents of not-so-important user activity, while another component is trying to update user account details. If you would look at your shiny Elasticsearch metrics, everything will look fine since 10 million document updates will make the statistical effect of user account updates disappear. On the other hand, your users might not be that happy with the latency they observe while they are trying to update their accounts. Hence always expose extra application-level metrics for your Elasticsearch queries. While Elasticsearch metrics provide sufficient indicators for the overall cluster performance, they lack the domain-specific context.

Invest in CPU!

I cannot emphasize this enough: invest in CPU!. Now open your dashboards, look at the metrics of your most recent Elasticsearch hammering session in production, and tell me whether you are disk I/O, memory, or CPU bound? I am not telling you to use a 32-core machine with an inferior SATA disk drive and 8 GB of memory. Rather what I am talking about is this: get a decent machine with sufficient memory and SSD (you might want to checkout NVMe cards too depending on your disk I/O), after that invest in CPU. Judging from my past experiences, whether it is a write- or read-heavy load, CPU has always been our bottleneck.

Avoid writing custom Elasticsearch plugins

Once I had the opportunity to have the joy of pairing with a colleague to write an Elasticsearch plugin that exposes synonyms over a REST endpoint. That allowed us to query synonyms uploaded to an Elasticsearch cluster at runtime and manipulate it on-the-fly. After having 20 releases where we were convinced that there are no more concurrency bugs (actually, we were pretty convinced with the previous 19 releases too), it worked like a charm. Though the real suffering started when we tried to upgrade the Elasticsearch version supported by the plugin. In a nutshell, avoid writing custom Elasticsearch plugins, because…

Many Elasticsearch releases contain significant internal changes. It is highly likely the public APIs you base your plugin on will be hit by backward incompatible changes.
Getting concurrency right in an environment that you are not accustomed to can be daunting.
You need to tailor your deployment procedure to ship the plugin every time and everywhere it is needed. You cannot just use the vanilla Elasticsearch artifacts as is anymore.
Since your application depends on the specific functionality provided by the plugin, the Elasticsearch instances you run during integration tests also need to incorporate the plugin as well. Hence you cannot use vanilla, say, Docker images out of the box anymore.

Notes on "Reactive Programming in Spring 5"

2019-01-26T20:25:00Z

TL;DR – Already have your hands dirty with Spring? Wondering how reactive will play along with Spring and how they will shape each other? Read this book. You will get crushed under the load of information trying to get covered, but will have no regrets.

I tend to avoid reading books whose subject is about a certain library or tool, unless I think it will provide some insight into a certain domain in addition to the software’s internals. And as you may guess, many Spring-related books do not pass this test. Though Reactive Programming in Spring 5 was an exception due to following reasons:

Oleh Dokuka, one of the two authors of the book, is a contributor to many Spring projects, in particular to Reactor. He is further a pretty active figure in Reactor Gitter channel where he is serving a fantastic public service. I personally am really interested in hearing anything he would say about Reactive Streams in general.
Love it, hate it, but almost the entire Java train is fueled by Spring framework. In my day to day work, I am swamped into Spring too. If you would pay close attention, particularly after the recent SpringOne 2018, Spring is full steam moving towards a reactive interface supported by all its whistles and bells – RSocket, R2DBC, WebFlux and WebClient, Reactor, etc. Though sometimes it is really difficult to see how the entire machine is progressing given the excessive amount of sub-projects. I expected the book to provide a 2,000 ft overview to the roadmap of this beast.

Overview

If you are dealing with Spring while taking your daily dose of Java, read this book! The book manages to deliver an outstanding coverage of both reactive streams and its integration to almost all of the existing Spring components. In a nutshell, it lays out the following subjects in order:

fundamentals of reactive programming (observer pattern)
its historical development in the presence of RxJava, Reactive Streams, and Reactor
pull, push, and pull-push streaming
backpressure
theoratical foundations of reactive programming (Little’s and Amdahl’s laws)
WebFlux versus Web MVC
introduction to Reactor
reactive
- Spring Data
- Spring Cloud Stream
- Spring Cloud Flow
- Spring Boot
RSocket
testing
deployment to cloud

In conclusion, authors did a splendid job in compiling an exhaustive content covering almost every aspect of reactive programming in Spring 5. They possess an in-depth understanding of the material they are talking about and present it in a digestable form to the user. While this thick compilation has the potential of crushing the reader, it is definitely a rewarding read.

Pros

After finishing the book, a cover-to-cover marathon of 538 pages, I felt like I know everything there is to know about reactive programming in Spring.

The extent of the covered content was astonishingly huge. I knew about RxJava, Reactive Streams, and Reactor individually, but not how they relate to each other. The presented historical perspective cleared out the missing pieces of the puzzle for me. I really enjoyed how the prose builds up by examining the problem, introducing the solution, laying out caveats of certain approaches, and drawing real-world comparisons. They repeated this pragmatic pattern for almost any project (Reactor, WebFlux, Spring Data, Spring Cloud Stream, etc.) they introduced.

I was not expecting an in-depth Reactor content, though the authors delivered a perfect job here given the book’s main subject is not just Reactor.

Investigation of pull, push, pull-push streaming, stress of backpressure, and reinforcement of the content with some theory (Little’s and Amdahl’s laws) was pretty enlightening to read. While many would consider these subjects boring, I found it pretty interesting and practical.

Cons

The book is too long to a point of becoming exhausting. Many material (Vert.x, Ratpack integration, Reactive Streams TCK, transactions in distributed messaging systems, etc.) could have been left out and then the main juice would fit into ~300 pages. Even then, it is still long. I would have split the book into two: Project Reactor in Action and Reactive Programming in Spring 5. While reading, I also felt such a split as well.

While 90% of the book is pretty objective, certain subjects were presented with bias. Particularly, RSocket-vs-gRPC comparison and using message brokers over system-to-system communication chapters were really opinionated. (I further explained this problem in my notes shared below.)

The presented content is always excercised using Apache Kafka and MongoDB. I suppose this is due to the fact that these two are the only tools that have almost perfect reactive coverage by Spring family of projects.

Notes

Below I share my notes ranging from a grammar mistake to a code typo, incorrect or ambiguous information to a thought provoking know-how, practical tip, etc. I further classify them in 4 groups: corrections, improvements, questions, and other.

[p11] Shared processRequest() method is missing a return statement.
[p12] As a coroutine/fiber fanboy, I could not have passed through the following statement:

In some languages, such as C#, Go, and Kotlin, the same code [a simple inter-microservice communication over HTTP] might be non-blocking when green threads are used. However, in pure Java, we do not have such features yet. Consequently, the actual thread will be blocked in such cases.

I forgot how many times I needed to correct this false misbelief, but I will repeat it here one more time: For any language X compiling to JVM bytecode, if a Java standard library call is [thread] blocking, access to it in X is going to be blocking as well. Kotlin coroutines is no exception to this. Further every convenience X provides, say coroutines, except syntatic sugars, has already been at the exposure of Java via solid libraries like Kilim (since 2006) and Quasar (since 2013).

Additionally, the rules of the game will totally change in a couple of years after the release of Project Loom.
[p16] In Diagram 1.4, what is the difference between solid and dashed lines? I am surprised to see that the image is advertising Apache Kafka while the rest of the components are free from such specification.
[p18] In Diagram 1.5, texts are not readable. (Unfortunately, many of the image texts are not readable in the hardcopy. I don’t know about the e-book though.)
[p28] all cross-service communication is non-blocking anymore. → … is not blocking anymore.
[p46] the actual request processing continues until SseEnitter.complete() → … until SseEmitter.complete()
[p46] Justification of why @Async is necessary here is missing.
[p59] The count operator is pretty descriptive, it emits the only value … → … emits only one value …

[p62] Chapter starts with

public interface SearchEngine {
    List<URL> search(String query, int limit);
}

interface, improves it with

public interface InterableSearchEngine {
    Iterable<URL> search(String query, int limit);
}

and

public interface FutureSearchEngine {
    CompletableFuture<List<URL>> search(String query, int limit);
}

After piling up enough motivation, justifies the victory of Observables:

public interface RxSearchEngine {
    Observable<URL> search(String query, int limit);
}

As a reader, let me ask the obvious question: Why not using a Stream<URL> return type instead?

[p103] In the following explanation,

Note that ScheduledPublisher is effectively an infinite stream and the completion of the merged Publisher is ignored.

What does merged Publisher refer to? NewsPreparationOperator?
[p106] Unfortunately, building a proper test suit → … test suite
[p114] Rather than focusing this much on the TCK, which is most of the time more relevant for library authors compared to users, I wish the authors would have had spared some pages on pitfalls and patterns of processors, which is bread and butter of reactive stream users and sadly skipped in the previous chapter.
[p123-129] I don’t think the readers really care about reactive streams adjustments for Vert.x, Ratpack or MongoDB reactive streams driver. I find these pages irrelevant and distracting.
[p133] It is page 133 and the reader is newly getting exposed to Project Reactor. Sort of too late, I think.
[p133] Project Reactor, the most famous library in the reactive landscape, … Really? Putting aside I favor Reactor over RxJava, the IT community that I know of does not give me such an impression to back this claim up.
[p141] investigating its reach API. → … its rich API

[p149] Speaking of defer(), it allows retries, which is, I think, of uttermost importance mostly overlooked by many and aligns better with reactive streams semantic compliance. Consider the following code:

<T> Mono<T> f(Mono<T> mono) {
    Stopwatch stopwatch = Stopwatch.createUnstarted();
    return mono
            .doOnSubscribe(ignored -> stopwatch.start())
            .doOnTerminate(() -> LOGGER.log("{}", stopwatch));
}

versus the alternative below

<T> Mono<T> f(Mono<T> mono) {
    return Mono.defer(() -> {
        Stopwatch stopwatch = Stopwatch.createUnstarted();
        return mono
                .doOnSubscribe(ignored -> stopwatch.start())
                .doTerminate(() -> LOGGER.log("{}", stopwatch));
    });
}

Also this is a good spot to talk about assembly-vs-runtime overhead of operator chaining.

[p149] The word wrapping of paragraph starting with Summing this up, Project reactor allows … is broken.
[p150] …, we may still request Long.MAX_VALUE). → request Long.MAX_VALUE.
[p154] The first paragraph of the page (However, the described approach for defining subscription …) is an eye opener gem. Given many seasoned RS users fell into this trap at least once, a couple of paragraphs detailing these caveats would come really handy.
[p169] Speaking of Flux#flatMap() operator, concurrency parameter might have been slightly mentioned here.
[p171] Given we are executing doOn...() methods (doOnNext(), doOnComplete(), doOnSubscribe(), etc.) for their side effects, what does happen when they throw an exception?
[p173] Provide a couple of examples for log() operator, please! Many people, particularly coming from RxJava, do not know of its existence.
[p174] However, this allows forthe sending … → … allows for the sending …
[p174] In the Flux.create() example, I am puzzled about how does it handle the backpressure.
[p183] I think exponential backoff deserves an explanation here. What does it mean? Further, I would have used seconds rather than milliseconds in the example, since the latter does not expose the backoff periods clearly at first sight.
[p188] Flux<Integer> cachedSource = source.share(); Here and in the rest, all of the occurences of cachedSource should be replaced with sharedSource.
[p191] The used assembly phase term will be described in next chapters, hence using it here is sort of confusing. Further, the given explanation for transform() has still some room for improvement. Additionally, I would prefer to see a “when to use compose(), when to use transform()” guide.
[p196-197] There are 3 places where assemble-time is used, though it used to be referred as assembly-time”.
[p198] … passing Subscription through ever Subscribers … → … passing Subscription through Subscribers …
[p200] The last paragraph of p199 goes as follows:

THe importance of understanding this phase is that during runtime we may apply optimization that may reduce amount of signals exchange. [“the amount of signals exchanged”?] For example, as we are going to see in the next sections, we may reduce the number of Subscription#request calls and improve, therefore, performance of the stream.

Then the following tip opens the p200:

… the invocation of Subscription#request method causes a write to the volatile field that holds demand. Such a write is an expensive operation from computation perspective, so it is better to avoid it if possible.

Hrm… I thought the main overhead of Subscription#request calls was unnecessary individual requests which could have been batched. Compared to this, write to a volatile does not really sound like an overhead.
[p200] Diagram 4.8 is missing some explanation.
[p204] In Diagram 4.11, how is backpressure preserved given there is an external buffer employed by publishOn. Who maintains this buffer? Is it the task queue of the ScheduledExecutionService used under the hood?
[p206] Totally lost with the following sentence: Under the hood, subscribeOn executes the subscription to the parent Publisher into Runnable, which is the scheduler for a specified Scheduler.
[p207] In the shared code snippet, .map() and .filter() are missing ... as input arguments.
[p208] How does SingleScheduler handle blocking functions spoiling time-sensitive executions?
[p209] … so the execution is attache to … → … so the execution is attached to …
[p209] In the next page, it has been admitted that the shared code snippet does not demonstrate a good usage of ThreadLocals. So why not sharing a proper use of ThreadLocals?
[p211] I think Such design decision … have its own Context. part of the last paragraph needs a rewrite.
[p213] … bottom (with id initial) has empty Context at all. → … bottom (with id initial) has empty Context at all.
[p214] The preciding code show a … → The preciding code shows a …
[p215] … we mentioned another sample of such … → … we mentioned another example of such …
[p217] In Diagram 4.15, the box at the bottom should be titled subscribe() instead of flatMap().
[p225] Preview Online Code Files at the bottom looks to be a typo.
[p230] … the WebFlux module provides built-in backpressure support … Cool! But how does it do that?
[p254] … using the PasswordEncoder#matchs … → … using the PasswordEncoder#matches …
[p254] In the shared snippet, which scheduler executes the map(p -> passwordEncoder.matches(...)) line? Netty I/O loop thread? If so (or some other scheduler of similar importance), is it wise perform passwordEncoder#matches here?
[p255] In the shared code snippet, isn’t flatMap(isMatched -> ...) should be replaced with map(isMatched -> ...)?
[p278] else if(resposne.statusCode() == EXPECTATION_FAILD) { → else if(response.statusCode() == EXPECTATION_FAILED) {
[p267] … template has a placeholder, dataSource, … → … template has a placeholder, playList, …
[p273-274] All apperances of Albom and albom should be replaced with Album and album, respectively.
[p278] As we can observe from the preciding diagram, with an increase in parallelization, the throughput of the system starts becoming slower and slower. Hrm… Actually the throughput keeps on increasing, though the gain is slowing down. Does it mean the increase in throughput?
[p281] The definitions of contention and coherence are left pretty ambiguous. A couple of practical examples would come really handy.
[p292] Diagram 6.16 misses the legend. The caption can be reworked to fix this as follows: WebFlux (dash) versis WebMVC (plus) throughput… This applies to the rest of diagrams with multiple lines in the following pages.
[p301-303] I would rather use a bar chart for performance figures rather than a table.
[p308] Diagram 6.25 looks like a mspaint.exe 101 assignment submission.
[p311] … the system has four central components … → … the system has three central components …
[p349] … EclipseLink, Spring Data JDBC, and Spring Data JDBC, … → … EclipseLink, Spring Data JDBC, and Spring JDBC, …
[p356-357] All occurences of updatedBook... should be replaced with updateBook....
[p379] What is the point of Distributed transactions with the SAGA pattern here?
[p405] In Diagram 8.1, there is plenty of space that could have been used and the texts are so small to the point of becoming unreadable.
[p418] This page concludes the Scaling up with Cloud Streams chapter by announcing its triumph over previously mentioned service-to-service architectures. I sadly find this conclusion pretty biased and far from reflecting the reality. There is no silver bullet for such problems and as has always been the case, it depends. Further, almost all of the listed so-called improvements to message broker-based solutions are either as is or with similar approaches applicable to service-to-service architectures as well. It is also surprising to see no mention of practical hurdles one need to overcome while scaling brokers too. You can easily Google and find dozens of Kafka horror stories. For a book that spent chapters on backpressure problems, the question of how is it affected by the delegation of the communication to a broker is totally ignored. Given how objective and well-equiped the authors were throughout the entire book, I am really amazed to see such a hype-oriented conclusion at the end of this chapter.
[p425] In the shared code snippet, used messagesStream and statisticStream are neither described, nor defined.
[p430] For a book published in October 2018, it is better to not limit the list of cloud providers offering serverless deployments with AWS Lambda. To the best of my knowledge, GCP, Azure, and many others provide similar functionalities.
[p435] ClassLoad → ClassLoader
[p437] It is noted that authors escaped < and > characters in the URL they pass to curl with %3C and %3E, respectively. Though, AFAIK, curl already does that if you pass the URL in quotes. Isn’t it?
[p440] … represent independent ClassLoaders. → … represent independent ClassLoaders.
[p441] implements Function<Payment, Payment> → implements Function<Payment, PaymentValidation>
[p441] public Payment apply(Payment payment) { ... } → public PaymentValidation apply(Payment payment) { ... }
[p448] A dashed arrow is missing from Service B’s request(1) line to Service A’s request(10) line.
[p449] Page contains a pretty well wrapped comparison of Reactive Streams versus TCP for flow control. Given HTTP/3 is almost ready to be served and is employing UDP rather than TCP, a Reactive Streams versus HTTP/3 comparison would be really thought-provoking.
[p450] request(3) under Service A’s adapter is, I believe, mistakenly placed.
[p454] In the shared code snippet, isn’t Flux.interval(Duration.ofMillis(100)) too harsh for service discovery?
[p456] I am aware of the fact that the design of gRPC incurs serious flow control problems, though in a section where you compare contenders like RSocket and gRPC, there is no place for such a vague and ungrounded statement:

Looking at the preciding code, we may get the feeling that gRPC, along with asynchronous message passing, gives backpressure control support as well. However, that part is a bit tricky.

Is backpressure is not handled at all or what? One should not pass over such a claim by hand waving.
[p457] public static class TestApplication → public class TestApplication
[p457] The RSocket endpoint is annotated with @RequestManyMapping(value = "/stream1", mimeType = "application/json"). Given RSocket is a wire protocol, what is the purpose of the provided path and MIME type here?
[p461] The summary repeats the triumph of message brokers over service-to-service communication one more time. Again, this is a really subjective statement. There are many advantages and disadvantages to each approach. Announcing a winner is a pretty dangerous generalization.
[p471] Shared StepVerifier.withVirtualTime() trick only works unless you use a custom scheduler in interval() and timeout() calls.
[p478] Here authors share a hack to work around the problem of Spring Boot 2.0 WebClient not providing the mocking support for outgoing HTTP interactions. I would rather share this trick in a Stack Overflow post, because the readers of the book will take it for granted, and rather share the aforementioned WireMock approach.
[p480] Testing WebSocket… Again, this is a hack rather than a solution. I would not share it in the book.
[p491] Diagram 10.1 is totally irrelevant to the content of the page.
[p492] Some texts in Diagram 10.3 are not readable.
[p503] … memory usage by regions, GC pauses, treads count … → … memory usage by regions, GC pauses, thread counts …
[p510] It is wise to note that Spring Boot Admin is not an official Spring project.
[p514-515] No mention of either GCP or App Engine.

Using Elasticsearch as the Primary Data Store

2018-11-14T07:54:00Z

The biggest e-commerce company in the Netherlands and Belgium, bol.com, set out on a 4 year journey to rethink and rebuild their entire ETL (Extract, Transform, Load) pipeline, that has been cooking up the data used by its search engine since the dawn of time. This more than a decade old white-bearded giant, breathing in the dungeons of shady Oracle PL/SQL hacks, was in a state of decay, causing ever increasing hiccups on production. A rewrite was inevitable. After drafting many blueprints, we went for a Java service backed by Elasticsearch as the primary storage! This idea brought shivers to even the most senior Elasticsearch consultants hired, so to ease your mind I’ll walk you through why we took such a radical approach and how we managed to escape our legacy.

Before diving into the details, let me share a 2,000ft overview of an e-commerce search setup that will help you to gain a better understanding of the subjects discussed onwards. Note that this simplification totally omits a nebula of incorporated caching layers, systems orchestrating multiple search clusters, queues with custom flush and replay functionalities, in-place resiliency mechanisms, services maintaining deprecated search entities to avoid getting ranked down by bots due to 404s, circuit breakers, throttlers, load balancers, etc. But it is still accurate enough to convey the general idea.

The Search
The ETL
- Real-time Content Stream
- Configuration Stream
Operational Overview
- Configuration Mutations
- Configuration Predicates
The Old ETL
The Battle of Storage Engines
- Benchmark Setup
- Benchmark Results
The New ETL
- The Primary Storage: Elasticsearch
- The Configuration DSL: JSON and Groovy
Conclusion
Acknowledgements
F.A.Q

The Search

[Before going any further, I want to take this opportunity to align you on what exactly I do mean by search. I hope this will help you to better wrap your mind around the ultimate consumer of ETL. That being said, feel free to skip this section and directly jump to the ETL deep dive in the next section.]

Many people tend to make the mistake of having a narrow view on search at e-commerce and confining its use case to a mere term scavenging in a mountainous stack of product attributes. While this statement holds to a certain extent, it resembles a cherry located at the tip of an iceberg. (In Varnishing Search Performance presentation, I tried to summarize how difficult it can get just to add a caching layer between your search logic and backend.) There are books written, university lectures offered, and computer science branches dedicated on the matter. But let me try to briefly elaborate this from an engineering standpoint.

What is search anyway?

If I would try to give a general, but far from complete, overview, it enables one to

search for a term in hundreds of product attributes, where matching and ranking are curated with directly or indirectly available consumer (are you a PS4 owner searching for the newest “Call of Duty”?) and relevance (you probably meant a band by typing “The Doors”, which is irrelevant for “Doors & Windows” department) contexts,
browse (basically a search without a term) in thousands of categories with similar ranking mechanics used in search aforementioned,
beam up directly to a certain product or category given the input matches with certain patterns (EAN, ISBN, ISSN, etc.) or merchandising rules (any syntactic and/or semantic combination of “wine glasses” should end the flow in a particular department, etc.),
implicitly trigger multiple searches under the hood (e.g. narrowing down to a lower category or widening up to a higher category, etc.) to enhance the results,
and decorate every listing with faceting (you probably want to see “Capacity” facet rather than “Shoe Size” while searching/browsing in “Harddisks”) support.

Who/What is using search?

This is a big debate. But I know a handful of certain consumers:

Customers: People who search and buy goods. They look harmless, until one gets exposed to them on a Black Friday where they work hand to hand in masses to DDoS the entire infrastructure.
Bots: They periodically (a couple of times a day at most, as of the date of this writing) try to digest your entire catalog into their system for two main purposes:
- Integrate the catalog into their own search engine (that is, Google),
- Tune their pricing strategy (that is, competitors)
The worst part of handling bot traffic is you cannot always throttle them (for instance, Google takes into account website latencies for rankings) and you need to make sure they do not harm the customer traffic. Food for thought: Imagine your customers swarming at your shop at Christmas Eve and Google decided to spider your entire catalog with thousands of requests per second.
Partners: Your business partners can also scan your catalog periodically to integrate into their own systems. (Fun fact: Some even require a daily Excel export.) One can classify them as bots only interested in a subset of the data.
Internal services: Last time I counted, there were 20+ internal services using search to enhance their results in addition to the users I listed above. Their usage can constitute up to 50% of the traffic.

In the case of partners and internal services, one might argue why do they need the search data rather than directly accessing the raw product attributes and offers. The answer is simple: They also use additional attributes (e.g., facets, categories) incorporated at the ETL pipeline. Hence, rather than exposing the internal ETL system to them, it is more convenient to manage them at the search gateway which is known to have battle-tested scalability and resiliency measures.

What about performance?

As decades-long experience in this domain points, making search 10ms faster can yield millions of euros extra revenue depending on the scale of your business. Unfortunately, this equation works the other way around as well. Hence, you are always expected to perform under a certain latency and above a certain throughput threshold.

How volatile is the content?

Very, very, very volatile! I cannot emphasize this enough and I believe this is a crucial difference that puts e-commerce search apart from Google-like search engines – recall the conflict between Google and Twitter for indexing tweets. Maybe examples can help to convey the idea better:

A product might have multiple offers (bol.com offer, partner offer, etc.) featuring varying properties (pricing, deliverability, discounts, etc.) where both offers and/or their properties are highly volatile. The offer might run out of stock, the price might change, etc. While customer-facing web pages are enhanced with the most recent data at runtime, search index might lag behind and provide an eventually consistent view. The volatility in this context might range from seconds to months. On prime time, e.g. on Valentine’s Day, you don’t want your search engine to return gift listings that ran out of stock a couple of seconds ago.
Your manual (triggered by shop specialists) and automated (artificial intelligence, machine learning driven) processes can alter the category tree, add new facets, tune the exposure of existing facets, modify the search behavior (e.g., flows triggered by merchandising rules), add context sensitive (e.g. category-dependent) thesaurus entries, synonyms, introduce new rankings, etc. These changes might necessitate the update of millions of documents retroactively.

This volatility debate will take a prominent role while deciding on the architecture of the next ETL pipeline, which I will elaborate in a minute.

The ETL

In the domain of search at e-commerce, ETL denotes the pipeline where the input is a multitude of information sources (product attributes, offers, discounts, rankings, facets, synonyms, thesaurus entries, etc.) and the output is the denormalized input constituting search-ready documents optimized for search query performance. Wait a second? If an ETL pipeline just delivers some optimization purposes, doesn’t this sound like that one can have a search without it? Sorta… That is indeed possible to a certain extent. If we would put the details aside for a moment, we can roughly compare the two approaches as follows:

Strategy	Advantages	Disadvantages
Without ETL	Every change in the input sources take immediate effect. (Hence, almost zero index time cost.)	Latency and throughput hurts dramatically due to necessitated join and enrich operations on input sources at query time.
With ETL	Since all potential data to satisfy search requests has already been baked into the index, search necessitates the least amount of effort to satisfy a request at query time.	Every change in the input sources will necessitate pre-processing affecting a multitude of products ranging from a couple to millions.

Put another way, ETL is all about the trade-off between index- versus query-time performance. In the light of all these and given

our existing ETL was functionally comprehensive enough,
query time performance of Elasticsearch has already been suffering due to faceting, internally triggered queries, etc. to an extent external caching becomes a necessity,
and search latency has a big impact on the revenue,

we took the thick ETL pipeline path.

But what is this ETL pipeline really? What does it literally do? In order to answer these questions, let me focus your attention to the input sources going into the ETL pipeline:

[GPC stands for Global Product Classification, which is de facto commercial categorization of goods varying from a car to a litre of milk.]

These two input sources, content and configuration, feature two totally different execution patterns framing the functional requirements of the potential ETL solutions, hence, play the uttermost critical role in justifying the plan we picked. Let’s examine them further:

Real-time Content Stream

Here the ETL pipeline listens from more than a dozen queues for updates ranging from product attributes to offers, offer-specific discounts to rankings, etc. all formatted in JSON. Fortunately, each real-time content stream message triggers a single product update. Let me exemplify this with a case: when disk_capacity_bytes attribute of a product changes, we

first fetch the relevant document from the storage,
update its disk_capacity_bytes attribute,
apply configuration(s) matching with the last state of the updated document,
and persist the obtained result back.

There are some concerns need to be addressed here:

This is a pretty CPU intensive operation. Configurations, in essence, are rules in the form of (predicate, mutation) pairs defined via business-friendly screens by shop specialists. When an attribute of a document gets updated, this change might be of interest to many configurations which are determined by performing an inverse lookup on tens of thousands of configuration predicates (e.g., attrs.disk_capacity_bytes != null) matching with the last state of the document. Later on mutations (e.g., doc.disk_capacity_gigabytes = attrs.disk_capacity_bytes / 1e9) of the found configurations are executed to let them shape the document according to their needs.

This innocent looking procedure sneakily introduces two critical issues under the hood:
1. How would you represent the configuration predicate such that you can match them against the content?
2. How would you represent the configuration mutation such that you can execute them against the content?
And it goes without saying, both concerns aforementioned need to be engineered efficiently. You are expected to repeat this procedure on each message JSON of the real-time content stream where the traffic is in the order of millions per day.

As a concrete configuration example consider the following: You have two “Disk Capacity” facets defined by business: one for computers, one for smart phones departments. The first one translates the disk_capacity_bytes into a disk_capacity_terabytes attribute which is defined to be exposed when category == "computers" and the second translates into a disk_capacity_gigabytes attribute which is defined to be exposed when category == "smart phones". Here both configurations are executed when the attrs.disk_capacity_bytes != null predicate holds.
This operation needs to be performed atomically. Two concurrent operations touching to the same product should not result in a corrupt content.

Configuration Stream

Configurations are the rules defined via business-friendly screens. There modifications done by shop specialists are published in snapshots when they think the changes grow into a stable state that they are ready to be exposed to the customer. Each published configuration snapshot ends up serving three purposes:

search gateway uses it to determine how to query the search index,
ETL pipeline uses it to process the real-time content stream,
and ETL pipeline retroactively updates the documents that are potentially affected.

While the first two are relatively cheap operations, the last one is the elephant in the room! This is the first time in our beautiful tale described so far that we need to propagate a change to millions of documents. Let me further explain this in an example:

Let’s consider that the following category definition:

if (attrs.gpc.family_id == 1234 && attrs.gpc.chunk_id == 5678) {
  doc.category = "books"
}

is modified as follows:

if (attrs.gpc.family_id == 1234 && attrs.gpc.chunk_id == 0xDEADBEEF) {
  doc.category = "AWESOME BOOKS"
}

Sir, you are in trouble! As the very ETL pipeline, what you are expected to deliver is to

find products that are matching with the old predicate,
revert the changes of the old configuration mutation by removing books from the category field,
find products that are matching with the new predicate,
and apply the changes of the new configuration mutation by adding AWESOME BOOKS to the category field.

This easier said than done operation contains many implicit concerns:

ETL needs to avoid removing books from the category field if there are rules, other than the changed one, adding books to the very same category field. There are two ways you can approach to this:
1. With every value added to a field, store a meta information pointing to the rules associated with that value. These back-tracking pointers optimize the check whether a value can be removed or not, with the cost of maintaining them in an ocean of values.
2. After removing every value, put the product back into the ETL pipeline just like handling products in the real-time content stream. If there are any rules, other than the changed one, adding books to the very same category field, they will kick in. This simple approach comes with the cost of a CPU intensive and unfortunately mostly redundant processing.
Given that configuration predicates are allowed to access any field, how would one represent a predicate and translate this into an ETL storage query filter that performs well? (You would not want to scan the whole data set for each predicate that is changed, right? Well… depends.)

Let’s first discuss the representation of predicates issue, which was also a concern in the real-time content stream processing. Here you might first fall into the trap of whitelisting the operators (==, !=, >, >=, <, <=, ~=) and the content attributes (attrs.gpc.family_id, attrs.gpc.chunk_id, attrs.disk_capacity_bytes, etc.) that are allowed in configuration predicates. While whitelisting operators is fine, whitelisting the content attributes implies that the ETL pipeline, the configuration administration GUIs, etc. all needs to have the knowledge of this whitelist which strictly depends on the structure of the real-time content stream message structures. Whenever the message structures change or you want to add a new attribute to this whitelist, both happen a couple of times every year, you need to propagate this to many components in your service milky way and perform a deploy without downtime.

What about translating these predicate representations into efficient ETL storage query filters? Let’s take the simplest approach: Represent each attribute with a separate field. Then let me ask you the following questions:
1. If you would opt for using an RDBMS, you can represent attributes by columns and create an index for each individual column. (Ouch!) Thanks to the half-century battle-tested RDBMS literature, the database can easily optimize and perform an index scan for the constructed queries:
```
SELECT ...
  FROM content
 WHERE attrs_gpc_family_id = '1234'
   AND attrs_gpc_chunk_id = '5678'
```
  That being said… What if you hit to the maximum column count limitation? (Yes, we did!) Further, what about attributes that are list of objects:
```
{
  "authors": [
    {
      "fname": "Volkan",
      "lname": "Yazici"
    },
    {
      "fname": "Lourens",
      "lname": "Heijs"
    }
  ]
}
```
  You definitely cannot store these in a single column and still query each individual component. Ok, then you can normalize the data as follows:
```
SELECT ...
  FROM content,
       attribute AS a1,
       attribute AS a2
 WHERE a1.content_id = content.id AND a1.name = 'gpc_family_id' AND a1.value = '1234'
   AND a2.content_id = content.id AND a2.name = 'gpc_chunk_id'  AND a2.value = '5678'
```
  So far so good. But… In a matter of months, you will need to start partitioning tables and maybe even move certain partitions into separate database instances to maintain the latency under a certain threshold. (Yes, we did this as well!) But this never-ending database structure optimization more and more feels like you are inventing your own distributed database using a plain RDBMS. Does this really still need to be this way in 2018?
2. If you would opt for using MongoDB, like using an RDBMS, you still need create an explicit index on each (whitelisted) field. For filters involving multiple fields (e.g., attrs.gpc.family_id == 1234 && attrs.gpc.chunk_id == 5678), MongoDB query optimizer can purpose individual field indices via index intersection. That being said, our experience with this feature has not been very pleasant.
  
  The issue where attributes might contain list of objects is not a problem for MongoDB.
3. If you would opt for Google Cloud Datastore, you will need to create explicit indices for each potential filter combination and order matters! Yes, you read that right! Let me exemplify this bizarre situation. If you have configurations with the following predicates:
  - attrs.gpc.family_id == 1234
  - attrs.gpc.chunk_id == 5678
  - attrs.gpc.family_id == 1234 && attrs.gpc.chunk_id == 5678
  - attrs.gpc.chunk_id == 5678 && attrs.gpc.family_id == 1234
  you need to define 4 different indices! Ouch! This in its own was a Datastore show stopper for us.
4. If you would opt for Elasticsearch, all fields are indexed by default and you can use them in any combination! Yay! No need for whitelisting! And similar to MongoDB, Elasticsearch also allows querying list of objects, you just need to declare them explicitly as nested. If you don’t even want to worry about that, you can add a dynamic mapping template to make each object nested by default. Following is the index mapping you can use for that purpose:
```
{
  "date_detection": false,
  "dynamic_templates": [
    {
      "strings": {
        "match_mapping_type": "string",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "objects": {
        "match_mapping_type": "object",
        "mapping": {
          "type": "nested"
        }
      }
    }
  ]
}
```
  Above mapping also disables analyzing the fields of type string, since we are not interested in performing fuzzy queries. Clearly, date detection is disabled for similar reasons.
  
  These being said, Elasticsearch is known to suffer from deteriorating query performance over time when exposed to high update rates.

Operational Overview

So far we examined the current ETL setup with concrete examples for several cases. We broke down the system into its individual input sources and detailed their implications on certain architectural decisions. Let’s wrap up this mind-boggling details into operational abstractions:

Given these operational abstractions, let me summarize the constraints the configuration components (predicate and mutation) imply.

Configuration Mutations

If you would recall, configuration mutations were simple document enhancement instructions that I exemplified as follows:

doc.category = "books"

Here doc is a dictionary denoting the ETL’ed document source and mutation “adds” books value to its category field. This (for simplification purposes, JavaScript-employed) innocent looking expression can (and does!) go to unintended extents:

if (attrs.suitable_for_month <= 2) {
  doc.childhood_stage = "newborn";
} else if (attrs.suitable_for_month <= 12) {
  doc.childhood_stage = "infant";
} else if (attrs.suitable_for_month <= 48) {
  doc.childhood_stage = "toddler";
}

The choice of the mutation DSL employed is expected to deliver the following requirements:

It “must” support JSON input and output for the real-time content stream. (See step B4 in the figure.)
It “should” support ETL storage input and output for the configuration snapshot stream. (See step A4 in the figure.)

The reason that the latter functionality marked as optional is that the ETL pipeline can also retrieve these documents in raw from the storage, convert them to JSON, execute mutations, and persist them back again – assuming data integrity is provided by other means, e.g., transactions, retries powered by compare-and-swap operations, etc.

Configuration Predicates

Configuration predicates were simple conditions restricted to use a whitelisted set of operators (==, !=, >, >=, <, <=, ~=) supporting grouping:

attrs.gpc.family_id == 1234 && attrs.gpc.chunk_id == 5678

Similar to mutations, the choice of the predicate DSL used is expected to deliver the following requirements:

It “must” support JSON input for the real-time content stream. (See step B2 in the figure.)
It “should” support ETL storage input for determining the affected documents by the configuration snapshot delta. (See step A4 in the figure.)

We relaxed the latter constraint since one can very well prefer to put the entire stored document collection (Ouch!) back into the ETL pipeline, process them, detect the changed ones, and persist the updates. This approach has certain assumptions though:

We don’t need to perform this too often. That is, the frequency of configuration snapshots are relatively low, e.g., max. a couple of times a day.
The snapshot deltas affect a significant percentage of the entire collection to an extent that the advantage of finding and processing only the affected documents diminishes.

Given you still need to make a back of the envelope calculation on your cloud bill for each approach, our years of statistics in the ETL snapshot configuration point that most of the time snapshot deltas affect at most 5% of the entire collection and the average is less than 1% – thanks to the incremental updates carried out by shop specialists. Hence, performing a complete ETL a couple of times a day feels like overkill and hurts the engineer within you.

The Old ETL

The old ETL was a single Oracle database where the configurations were modeled in PL/SQL. Since the configuration abstraction language was the very same language the database uses itself, executing mutations and predicates was effortless. Hail SQL injection as a feature! Though this came with some notable costs:

Using PL/SQL within the abstraction model created both functional and financial vendor lock-in. The functional deficiency (incompetent expressiveness, leakage of PL/SQL to irrelevant components) obstructed many innovations over the years, where it became more and more difficult as time passed. Additionally, it constituted a significant obstacle for migrating the service to the cloud. Its financial aspect was negligible at the scale of bol.com.
Rolling back changes of an updated configuration mutation is quite a PL/SQL engineering endeavor to implement in practice. This difficulty, spiced up with the insufficient logging, testing, debugging, profiling, etc. utilities, drew programmers back from taking this path. Hence, there was a 12+ hours long complete ETL run every night for configuration snapshot deltas. This beast tamed by an experienced couple of engineers has a reputation to have frequent hiccups and make bugs really difficult to debug, find, and reproduce, let alone fix!

In its previous incarnation, the content attributes were stored in <id, content_id, key, value> normalized form. This approach started to suffer from efficiency aches in the hinges pulling the ETL’ed data to the search index. Back then hired Oracle consultants examined the usage and recommended to go with a denormalized structure where each attribute is stored as a column. In addition to temporarily bandaging up the efficiency related wounds, this allowed DBAs to let their imaginations go wild to map the attributes to columns. Recall the attributes composed of objects I mentioned above? Special characters were used to create such multi-value attributes, which was pretty much (to put it mildly) unpleasant. But the killer bullet came in the form of a six-inch punch referred as the maximum allowed column count limit. But isn’t engineering all about how hard you can get it and keep moving forward? Yes, comrade! We thought so and used a single binary XML column to store attributes, queried them using Oracle XPath toolbox, escaped attribute values, finally concatenated them into SQL strings that are eventually executed, and for sure crossed our fingers.

There are a couple of important details that I could not manage to cover in the above war diary without spoiling the coherency. Let me drop them here in no particular order:

Task parallelization is pretty difficult in PL/SQL. We tried patching this hole via internal Oracle AQs, but I am not really sure whether it improved or worsened the state.
In a database procedure that is expected to run for 12+ hours, Murphy’s law works flawlessly. Anything that can go wrong, did, does, and will go wrong. We wisely(!) engineered the system to persist its state at certain check points constituting retriable handles to invoke when you come in the morning and see that the ETL crashed.
The number of moving components necessitated the use of a proprietary scheduling tool supporting Oracle. The schedule was glued with bash scripts, designed in a proprietary development environment only available for Windows, and rolled out on Oracle machines running GNU/Linux. Neither GNU/Linux, nor Windows using developers were fond of this situation.
Due to the high cost of a failing ETL, business also did not feel empowered to change and/or commercially optimize it easily. This was a pretty demotivating issue affecting both technical and business people need to work with it.

Enough blaming the former engineer. We need to get our facts right. The aforementioned PL/SQL giant was not rolled out in a day with a big bang. This more than a decade old ETL pipeline was developed with all the best practices and tooling available back then. The more you dive into its source code, navigate through commits of features spanning through years, it becomes easier to see what went wrong and where. Now you are able to realize the patterns that necessitated exceptional handling of certain features, of which many due to backward-compatibility with legacy systems that have already been deprecated or replaced by newcomers, exploded the complexity to unintended depths. Software development is never-ending progress and axioms you base your initial architecture on become invalidated in the course of time due to changing business needs. Aiming for infinite flexibility comes with an engineering cost as well, which might very well fall short of justifying such an expense. One should also include the massive burst of data volume and its update frequency into this list. I personally think the old ETL pipeline and its engineers did a fantastic job. The tool served its purpose for more than a decade and harvested an immense amount of lessons for its successor. I would be more than happy if we as a team can also achieve to deliver such a long living product.

The Battle of Storage Engines

Given our functional requirements, we evaluated a couple of different ETL pipeline storage solutions which I hinted to earlier. Following is the feature matrix of each candidate:

Storage Solution	Distributed	Sharded	Required Indices	Integrity Measure
PostgreSQL	No	No	One¹	Transactions
PostgreSQL (partitioned)	No	Yes²	One¹	Transactions
MongoDB	Yes	Yes³	Some⁴	Compare-and-swap⁵
Elasticsearch	Yes	Yes	None	Compare-and-swap⁶

¹ PostgreSQL jsonb index covers all fields.
² PostgreSQL partitioning is not sharding in distributed sense, but still serves a similar purpose.
³ MongoDB sharding requires manual configuration.
⁴ MongoDB requires an explicit index for each whitelisted field allowed in ETL configuration predicates.
⁵ MongoDB updateMany() or findAndModify() can be leveraged for the desired integrity.
⁶ Elasticsearch _version field can be leveraged to implement a compare-and-swap loop.

Benchmark Setup

For the benchmark, we populated each store with 33 million JSON documents of which each weighs an average size of 2.5KB. One of the contrived fields in the document is search_rank. Later on, a file consisting of 6 million distinct <id, search_rank> pairs is streamed in batches of size 1000. For each batch, we first fetch the old search_ranks associated with the ids and then bulk update these with the new search_ranks. In this scenario, what we tried to emulate is a bulk update triggered by a configuration snapshot delta, which is the most storage performance demanding operation in the ETL pipeline.

Used test bed is a cluster composed of 6 dedicated machines with the following specifications:

CPU: 16 core Intel Xeon E5-2620 v4 @ 2.10GHz
Memory/Swap: 128GB/16GB
Disk: 375GB (Intel P4800X Performance NVMe PCIe SSD)
Kernel: 3.10.0-693.1.1.el7.x86_64

We further configured each store as follows:

PostgreSQL: Just one PostgreSQL 9.6.10 instance containing a single <id, content> table where content is of type jsonb. Benchmark configured to update only the search_rank attribute of the content column.
PostgreSQL (partitioned): Same as above, but the content table is partitioned into 10 tables.
MongoDB: MongoDB 3.6 with the following configurations:
```
systemLog.destination: file
systemLog.logAppend: true
processManagement.fork: true
storage.engine: wiredTiger
security.authorization: enabled
replication.oplogSizeMB: 9216
```
Note that sharding is not enabled. (More on this later.)

Similar to PostgreSQL setup, benchmark configured to update only the search_rank attribute of documents.
Elasticsearch: Elasticsearch 6.3.0 with the following JVM flags:
```
-Xms30g
-Xmx30g
-Xss256k
-XX:NewRatio=3
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintClassHistogram
-XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
```
Here JVM heap size is set to 30G due to compressed OOPs limitation.

Different from PostgreSQL and MongoDB setups, where only the search_rank attribute is updated, Elasticsearch benchmark is configured to update the entire document. While this overkill is subject to hammer Elasticsearch way heavier (since Elasticsearch will create quite some garbage segments waiting to be merged and making every object nested worsens the case even more) than other stores, it is more strategically aligned with how we want to use it in the future.

Benchmark Results

Below you will see the results of the benchmark for only MongoDB and Elasticsearch. The reason PostgreSQL results were omitted is no matter what kind of optimization we throw at it, the benchmark always took more than 2 hours, regardless of partitioning, whereas MongoDB and Elasticsearch took a couple of minutes.

Store	Conc.⁷	Latency	Total (s)	Fetch⁸ 75% (ms)	Fetch⁸ 99% (ms)	Fetch⁸ Max. (ms)	Update⁹ 75% (ms)	Update⁹ 99% (ms)	Update⁹ Max. (ms)
MongoDB	8	total	518	68	999	3380	64	2347	4153
	8	per batch		8	125	423	8	293	519
	16	total	526	71	3082	7905	68	5564	7955
	16	per batch		4	193	494	4	348	497
	32	total	518	61	6668	11465	98	10533	13784
	32	per batch		2	208	358	3	329	431
Elasticsearch	8	total	251	278	423	798	94	186	412
	8	per batch		35	53	100	12	23	52
	16	total	196	478	697	1004	141	266	410
	16	per batch		30	44	63	9	17	26
	32	total	175	951	1368	1515	214	331	828
	32	per batch		30	43	47	7	10	26

⁷ Number of concurrent batches.
⁸ Time it takes to fetch a batch.
⁹ Time it takes to update a batch.

Let me share some observations from the results:

Increasing concurrency improves Elasticsearch performance (up to 32 concurrent batches) but does not have much effect on MongoDB.
Elasticsearch rocked in performance even though it is hammered with the update of the entire document whereas MongoDB is just trying to update a single attribute. Using 32 concurrent batches, it took 175s and 518s for Elasticsearch and MongoDB, respectively, to complete the benchmark.
Elasticsearch yielded way more predictable performance figures compared to MongoDB. Note the difference between 75- and 99-percentile figures.
Elasticsearch segment merges were unexpectedly pretty stable during the runs, whereas we were anticipating it to become the bottleneck due to high update rate. But compare-and-swap loops played over _version fields allowed for the necessary data integrity without breaking a sweat.

At the time of testing, we initially were not able to enable sharding in MongoDB due to operational obstacles on our side. Though Elasticsearch results were such promising, to the point of even shocking the hired Elasticsearch consultants, we decided to go with it, of which we have years of production experience. If we would put the necessity of whitelisted configuration predicate fields problem aside – that is, required explicit indices on what can be queried – MongoDB could very well be a viable option as well.

But, really, why Elasticsearch has a reputation of not being recommended as a primary data store? I think it all started when the official project website years ago contained an explicit statement admitting that Elasticsearch is not intended to be used as a primary data store. Once, as the very owner of the project itself, you admit this fact, it is really difficult to convince people the other way around – even if the situation might have been improved. Later on, published Jepsen (an effort to improve the safety of distributed databases, queues, consensus systems, etc.) reports (one in 2014-06-15 using Elasticsearch 1.1.0 and the other one in 2015-04-27 using Elasticsearch 1.5.0) worsened the situation and this bad reputation disseminated over the web in the speed of light. While this tornado DDoS’ing the entire Hackernews, Proggit, etc. blogosphere with endless discussions in the form of “See? I told ya so!”, Elasticsearch team put up a Elasticsearch Resiliency Status page. There they started sharing (even up to today!) known resiliency problems, including the ones found in Jepsen reports, converting them into reproducable cases in GitHub issues, and tackling them one at a time. What else would qualify as a professional commitment if not this one? Again, these were all back in early 2015. Our Elasticsearch production deployments successfully managed to return with a victory from every battle front thrown at them. It did not always feel like a walk in the park. We had our hard times, though managed to overcome those and noted down the experience to the book of lessons learnt. Let me share some common practices from that collection:

Security: Elasticsearch does not provide any means of security measures (encryption, etc.) out of the box. We do not use Elasticsearch to store any sort of PII.
Transactions: Elasticsearch does not have transaction support. Though we work around it by performing compare-and-swap loops over the _version field.
Tooling: Elasticsearch tooling is… just a piece of crap. It doesn’t have a proper development environment – you are stuck to running a fully blown Kibana just to be able to use its arcane Console. Its Java client drags in the entire milky way of Elasticsearch artifacts as a dependency which is a JAR Hell time bomb waiting to explode. Further, the recently introduced high-level REST client leaks the Apache HTTP Client API models, etc. For the leaked models and transitive dependencies, there is nothing much you can do – you just learn to live with them. For IDE, you just keep a thick stack of HTTP request recipes using your favorite HTTP client, e.g., cURL, Postman, httpie, etc.
Documentation: Elasticsearch does not have documentation; PostgreSQL has documentation, MongoDB has documentation. What Elasticsearch has is a stack of surface-scratching blog posts served in the form of a documentation-like website. Elasticsearch also has an ocean of Stack Overflow and forum posts where you are allowed to swim at your convenience. That being said, one needs to admit that situation is improving over the time. (Yes, it was way worse!)
Resiliency: Yes, Elasticsearch can crash, just like another piece of software. In order to address these emergencies, in addition to hot-standby clusters, we take regular snapshots and persist the messages processed by the ETL pipeline to a separate storage providing efficient write and bulk read operations, e.g., PostgreSQL, Google BigQuery, etc. In case of need, we just restore from a snapshot and replay the necessary set of messages to recover the lost state.

Is Elasticsearch the perfect tool for the job at hand? Not really. But it is the one closest to that. We also know how to deal with each other – just like in any other relationship.

The New ETL

By taking into account the ETL pipeline concerns detailed in previous chapters, we derived a list of basic foundations that we aim to deliver:

The configuration DSL must be abstract enough to avoid ~~any~~ too much vendor lock-in. One must be able to represent configurations in this DSL such that applying these on a JSON and/or the underlying storage unit must be a matter of writing the necessary adapter classes.
The storage must allow the ETL pipeline to query the entire collection using any possible filter combinations allowed by the configuration predicate DSL. This is a crucial pillar in the design to enable real-time processing of every message, both content and configuration snapshot stream, without necessitating an ETL run over the complete collection which used to be the case in the old ETL pipeline.

Let me elaborate on how we addressed these deliverables.

The Primary Storage: Elasticsearch

The previous benchmark section already detailed the rationale behind employing Elasticsearch as the primary storage. It is distributed and sharded by default. It doesn’t require explicit indices on a whitelist of allowed configuration predicate fields – every field is allowed to be queried by default. It has no problems with querying fields containing a list of objects. It provides sufficient leverage for data integrity via compare-and-swap loops over _version fields. It is very efficient on bulk fetches and updates, which was totally unexpected for us. Last, but not least, it is our bread and butter in search and we have plenty of experience with it.

The Configuration DSL: JSON and Groovy

In the case of configuration DSL, we wanted to stop the plague of PL/SQL leakage all around the code base. For this purpose, we decided to go with the model depicted below.

Here we replaced SQL WHERE clauses, which were used to represent configuration predicates in the old ETL pipeline, with JSON describing the structure of the predicate. This new predicate representation resembling the Elasticsearch filters is translated to individual executors matching against either JSON (coming from the real-time content stream) or the storage engine, that is, Elasticsearch. Note that the way we used to represent the predicate is independent of medium (JSON, Elasticsearch, etc.) it is executed against such that we even implemented a MongoDB adapter at some point. An example configuration predicate JSON is show below:

{
  "type": "nested",
  "path": ["content", "attribute"],
  "filter": {
    "type": "and",
    "filters": [
      {
        "type": "nested",
        "path": ["gpc"],
        "filter": {
          "type": "and",
          "filters": [
            {
              "type": "equals",
              "path": ["family_id"],
              "value": "1234"
            },
            {
              "type": "equals",
              "path": ["chunk_id"],
              "value": "5678"
            }
          ]
        }
      },
      {
        "type": "nested",
        "path": ["authors"],
        "filter": {
          "type": "and",
          "filters": [
            {
              "type": "equals",
              "path": ["fname"],
              "value": "Volkan"
            },
            {
              "type": "equals",
              "path": ["lname"],
              "value": "Yazici"
            }
          ]
        }
      }
    ]
  }
}

As depicted above, we split the configuration mutation model into two abstractions: extension and functional extension. An extension is the simplest form of mutation that generally applies to more than 90% of the available configurations. It is basically a JSON object that is upon execution expected to be merged into the original source. A simple example is as follows:

{
  "category": "books"
}

Functional extensions are built to address complex configuration mutations. There we employed Groovy after experimenting with some other candidates, e.g., JavaScript (Nashorn, which is planned to be dropped), Python (Jython), Ruby (JRuby), etc. The main drivers for us to pick Groovy are as follows:

It supports direct access to Java data structures (e.g., java.util.Map) without any intermediate translations, hence has no problems processing thousands of mutations on a single core.
It is widely adopted to an extent that in the future we might opt for running it against the storage engine.
Its runtime performance is on par with the rest of the candidates.

That being said, the decision of Groovy creates a JVM vendor lock-in for the ETL pipeline, though we do not anticipate this to be a problem for at least the coming decade.

A sample functional extension is given below.

static Map<String, Object> extend(Map<String, Object> source) {
    def diskCapacityBytes = (long) source.get("disk_capacity_bytes")
    def diskCapacityGigabytes = diskCapacityBytes / 1e9
    source.put("disk_capacity_gigabytes", diskCapacityGigabytes)
    return source
}

Conclusion

Implementing an e-commerce search engine is a tough business. The part of the iceberg under the water level – that is, the ETL pipeline – is not less than that. In this post, I tried to share the lessons we piled up in the implementation and maintenance of our decade-old ETL pipeline and how we cultivated these to come up with something new. I attempted to explain how the choice for the configuration DSL and the used primary storage engine has the uttermost implication on the rest of the components of the architecture. Elasticsearch has already been serving us pretty well in the search gateway. Taking a step further and employing it in the ETL was a substantially unconventional idea that gave the shivers to every engineer involved in the decision. But the careful consideration and evaluation of potential candidates paid off: It worked! So when you visit bol.com next time, you will know that the Elasticsearch in the ETL pipeline – in addition to many other Elasticsearch using services involved – cooked that warm page for you seconds ago.

Acknowledgements

I would like thank to Berkay Buharalı, Lourens Heijs, William Leese, Almer S. Tigelaar, Leon Widdershoven, and Maurice Zeijen for their valuable feedback in bringing the post to its final form.

F.A.Q.

Here I will try to answer certain questions I received via Hackernews or e-mail.

Did you try tuning the PostgreSQL optimization knobs?

bol.com has plenty of databases supported by an army of skilled DBAs. During benchmarks, we collaborated with our PostgreSQL experts to continuosly tune the necessary knobs to get the best performance given our data size and access patterns. Hence, it wasn’t a tune once, run once operation, but rather a continuous effort to determine an optimal configuration.

How do you calculate `search_rank`?

For the benchmark purposes, we employed a deprecated signal (that is, search_rank) that we used to score the matched documents. In the new search gateway, that approach is replaced with a multitude of context-dependent signals combined at runtime. The answer to how does the computation of ranking signals work is out of the scope of this post. But in a nutshell, it is an in-house machine learning algorithm harvesting historical user interaction log. Handling of sudden or seasonal trends? That is a whole different game.

Have you ever presented this material in a conference setting?

Yes. I managed to squeeze the entire content into a 50 minutes long talk and had to opportunity to share it with people in

CNCML Vienna, 2019 (PDF, ODP)
ElasticNL (PDF, ODP)

Varnishing Search Performance

2018-02-17T20:42:00Z

This week bol.com hosted an Elastic User Group NL meetup titled bol.com: Changing the (search) engine of a racecar going 300 km/h. The abstract of the presentations were as follows:

Almost 2 years ago bol.com decided to move towards an Elasticsearch-powered search engine. But how do you approach such a project? Who do you involve and what do you need to (not) do? The engineers at bol.com would like to share their experiences about this migration, in 4 short talks.

And among those 4 short talks, I took the stage with Varnishing Search Performance.

Searching is peanuts. You setup your Elasticsearch cluster (or better find a SaaS partner) and start shooting your search queries against it. Well… Not really. If we put the biblical data ingestion story aside, it won’t take long to realize that even moderately complicated queries can become a bottleneck for those aiming for <50ms query performance. Combine a couple of aggregations, double that for facets of range type, add your grandpa’s boosting factors to the scoring, and there you go; now you are a search query performance bottleneck owner too! Maybe I am exaggerating a bit. Why not just start throwing some caches in front of it? Hrm… We actually thought of that and did so. Though it brought a mountain of problems along with itself, and there goes my story.

The slides are available in PDF and ODP formats.

Notes on "Netty in Action"

2018-02-09T20:34:00Z

Those who had priviledge to read my frustration chronicles on intra-microservice communication would easily recall me pointing my finger to Java Platform SE guys for not shipping a proper HTTP client. There my fury went to an extent calling it as one of the closest candidates for the billion dollar mistake. Unfortunately screaming out loud in a blog post does not give much of a relief, because it doesn’t take more than a month for me to find myself in precisely the same technical mudpot. Indeed after a couple of months later I wrote that post, I was chasing yet another performance problem in one of our aggregation services. In essence, each incoming HTTP request is served by aggregating multiple sources collected again over HTTP. This simple fairy tale architecture gets slaughtered on production by 200 Tomcat threads intertwined with Rx computation and I/O threads resting in the shades of a dozen other thread pools dedicated for so-called-asynchronous HTTP clients for aggregated remote services. And I saved the best for last: there were leaking TIME_WAIT sockets.

All of a sudden the question occurred to me like the roar of rolling boulders down a steep hill in a far distance: What is the lowest level that I can plumb a networking application in Java without dealing with protocol intricacies. Put another way, is there a foundational abstraction exposing both the lowest (channel with I/O streams) and highest (HTTP headers and body) levels that are in reach? I rode both Java OIO and NIO (that is, old- and new-I/O) horses in the past and fell off enough to learn it the hard way that they are definitely not feasible options in this case. The first attempt in the search of a cure in Google introduces you to Netty. If you dig long enough, you also stumble upon Apache Mina too. Netty is popular enough in the Java world that it is highly likely you are an indirect consumer of it, unless you are already directly using it. I was aware of its presence like dark matter in every single network application that I wrote, though I have never considered to use it directly. Checking the Netty website after dealing with crippled network applications at hand revealed an enlightenment within me: Hey! I can purpose this to implement some sort of RPC mechanism using Protocol Buffers in HTTP 2.0 request payloads! Though further investigation swipes the dust from the footsteps of giants who had followed the same path: Google (gRPC), Facebook (Nifty), Twitter (Finagle), etc. This finding while crushing my first excitement, later on left its place to the confidence of getting confirmed that I am on the right path.

I have always heard good things about both Netty and its community. I have already been sneakily following the presentations and Twitter updates of Norman Maurer, the Netty shepherd as of date. Though what triggered me for diving deep with Netty has become the following tweet:

Challenge accepted! First step is done. Next: Cover to cover study. pic.twitter.com/Gnfhbi6Ko0
— Volkan Yazıcı (@yazicivo) January 19, 2018

Norman Maurer has always been kind and encouraging to new contributors. So my plan is to turn this into a relation with mutual benefit: I can contribute and get tutored while doing that so.

Netty in Action

The book (2016 press date) is definitely a must read for anyone planning to use Netty. It lays out Netty fundamentals like channels, handlers, encoders, etc. in detail. That being said, I have got the impression that the content is mostly curated for beginners. For instance, dozens of pages (and an appendix) are spent (wasted?) for a Maven crash course, not to mention the space wasted by Maven command ouputs shared. This felt a little bit disappointing considering the existing audience of Netty in general. Who would really read a book about Netty? You have probably had your time with OIO/NIO primitives or client/server frameworks in the market. You certainly don’t want to use yet another library that promises to make all your problems disappear. So I don’t think you can be qualified as a novice in this battle anymore, and you are indeed in the search of a scalpel rather than a swiss army knife. Nevertheless, I still think the book eventually managed to succeed in finding a balance between going too deep and just scratching the surface.

Things that are well done

I really enjoyed the presented historical perspective on the development of Java platforms’ networking facilities and Netty itself. Found it quite valuable and wanted to read more and more!
Emphasis on ByteBuf was really handy. Later on I learnt that there are people using Netty just for its sound ByteBuf implementation.
Almost every single conscious decision within the shared code snippets are explained in detail. While this felt like quite some noise in the beginning, later on it turned out be really helpful – especially while manually updating ByteBuf reference counts.
Presented case studies were quite interesting to read and inspiring too.

Things that could have been improved

I had big hopes to read about how to implement an HTTP client with connection pool support. I particularly find this feature inevitable in a networking application and often not consumed wisely. Though there wasn’t a single section mentioning about connection pooling of any sort.
As someone who had studied Norman Maurer’s presentations, I was expecting to see waaaay more practical tips about GC considerations, updating socket options (TCP_NO_DELAY, SO_SNDBUF, SO_RCVBUF, SO_BACKLOG, etc.), mitigating TIME_WAIT socket problems, and Netty best practices. Maybe adding this content would have doubled the size of the book, though I still think a book on Netty is incomplete without such practical tips.
Many inbound requests trigger multiple I/O operations in a typical network application. It is crucial to not let these operatins block a running thread, which Netty is well aware of and hence ships a fully-fledged EventExecutor abstraction. This crucial detail is mentioned in many places within the book, though none gave a concrete example. Such a common thing could have been demonstrated by an example.

Notes

I always take notes while reading a book. Let it be a grammar mistake, code typo, incorrect or ambiguous information, thought provoking know-how, practical tip, etc. You name it. Here I will share them in page order. I will further classify my notes in 3 groups: mistakes, improvements, questions, and other.

[p19, Listing 2.1] Why did we use ctx.writeAndFlush(Unpooled.EMPTY_BUFFER) rather than just calling ctx.flush()?
[p21, Listing 2.2] Typo in throws Exceptio3n.
[p49, Section 4.3.1] The listed items
- A new Channel was accepted and is ready.
- A Channel connection …
are an identical repetition of Table 4.3.
[p60] CompositeByteBuf has the following remark:

Note that Netty optimizes socket I/O operations that employ CompositeByteBuf, eliminating whenever possible the performance and memory usage penalties that are incurred with JDK’s buffer implementation. This optimization takes place in Netty’s core code and is therefore not exposed, but you should be aware of its impact.

Interesting. Good to know. I should be aware of its impact. But how can I measure and relate this impact? Maybe I am just nitpicking, tough I would love to hear a little bit more.
[p77, Table 6.3] channelWritabilityChanged() method of ChannelInboundHandler… How come an inbound channel can have a writability notion? I would have expected an inbound channel to be just readable.
[p78, Section 6.1.4] Starts with some really intriguing paragraph:

A powerful capability of ChannelOutboundHandler is to defer an operation or event on demand, which allows for sophisticated approaches to request handling. If writing to the remote peer is suspended, for example, you can defer flush operations and resume them later.

Though it ends here. No more explanations, not even a single example, etc. A total mystery.
[p79, Table 6.4] read() method of a ChannelOutboundHandler… Similar to ChannelInboundHandler#channelWritabilityChanged(), how come an outbound channel can have a read method? What are we reading that is supposed to be already originating from us and destined to a remote peer?
[p79, Section 6.1.4] It goes as follows:

ChannelPromise vs. ChannelFuture Most of the methods in ChannelOutboutHandler take a ChannelPromise argument to be notified when the operation completes. ChannelPromise is a subinterface of ChannelFuture that defines the writable methods, such as setSuccess() or setFailure(), thus making ChannelFuture immutable.

Ok, but why? I know the difference between a Future and a Promise, though I still cannot see the necessity for outbound handlers to employ Promise instead of a Future.
[p84, Listing 6.5] While adding handlers to a pipeline, what happens in the case of a name conflict?
[p84] A remark is dropped on the ChannelHandler execution and blocking subject. Just in time! Though it misses a demonstration.
[p86, Listing 6.9] Again a read() method for the outbound operations of a ChannelPipeline. I am really puzzled on the notion of reading from an outbound channel.
[p94, Listing 6.13] What happens when a ChannelFuture completes before adding a listener to it?
[p95, Section 6.5] Last paragraph goes like this:

The next chapter will focus on Netty’s codec abstraction, which makes writing protocol encoders and decoders much easier than using the underlying ChannelHandler implementations directly.

Though next chapter focuses on EventLoop and threading model.
[p102, Listing 7.3] Speaking of scheduling Runnables to a channel’s event loop, what if channel gets closed before triggering the scheduled tasks?
[p103] Page starts with the following last paragraph:

These examples illustrate the performance gain that can be achieved by taking advantage of Netty’s scheduling capabilities.

Really? Netty’s scheduling capabilities are shown by using each function in isolation. Though I still don’t have a clue on how these capabilities can be purposed for a performance gain. This is a common problem throughout the book: The innocent flashy statement hangs in the air, waiting for a demonstration that shares some insight distilled by experience.
[p104, Figure 7.4] The caption of figure is as follows:

EventLoop allocation for non-blocking transports (such as NIO and AIO)

AIO? Looks like a typo.
[p107] Chapter starts with the following opening paragraph:

Having studied ChannelPipelines, ChannelHandlers, and codec classes in depth, …

Nope. Nothing has been mentioned so far about codec classes.
[p112] It is explained that, in the context of Bootstrap, bind() and connect() can throw IllegalStateException if some combination of group(), channel(), channelHandler(), and/or handler() method calls is missing. Similarly, calling attr() after bind() has no effect. I personally find such abstractions poorly designed. I would rather have used the staged builder pattern and avoid such intricacies at compile-time.
[p117, Listing 8.6] The 2nd argument to Bootstrap#group() looks like a typo.
[p120] Check this end of chapter summary out:

In this chapter you learned how to bootstrap Netty server and client applications, including those that use connectionless protocols. We covered a number of special cases, including bootstrapping client channels in server applications and using a ChannelInitializer to handle the installation of multiple ChannelHandlers during bootstrapping. You saw how to specify configuration options on channels and how to attach information to a channel using attributes. Finally, you learned how to shut down an application gracefully to release all resources in an orderly fashion.

In the next chapter we’ll examine the tools Netty provides to help you test your ChannelHandler implementations.

I have always found such summaries useless, since it is a repetition of the chapter introduction, and hence a waste of space. Rather just give crucial take aways, preferably in a digestible at a glimpse form. For instance, use EventLoopGroup.shutdownGracefully(), etc.
[p121] I suppose Unit Testing chapter used to come after Codecs in previous prints and the authors have moved it to an earlier stage to establish a certain coherence in the introductory chapters. Though, reading Codecs reveals that there is close to 70% overlap in content, which feels like a poorly structured flow. I see the value in authors’ attempt, though there is quite some room for improvement via tuning the break down of chapters.
[p124, Section 9.2.1] ByteToMessageDecoder is used before explained. (See my remark above.)
[p127] The following bullets
Here are the steps executed in the code:
1. Writes negative 4-byte integers to a new ByteBuf.
2. Creates an EmbeddedChannel …
is a repetition of the descriptions available in Listing 9.4.
[p138, Listing 10.3] Comma missing after Integer msg.
[p141] Why do MessageToMessage{Encoder,Decoder} classes do not have an output type, but just Object? How do you ensure type safety while chaining them along a pipeline?
[p142, Listing 10.6] Comma missing after Integer msg.
[p145, Listing 10.7] Constructor of MyWebSocketFrame is named incorrectly.
[p151, Section 11.2] I think Building Netty HTTP/HTTPS applications deserves its own chapter. And a very important subject is missing: connection pooling.
[p157, Listing 11.6] While building the WebSocket pipeline, which handler addresses ping/pong frames?
[p159, Table 11.4] The first sentence in the description of WriteTimeoutHandler is identical to the one in ReadTimeoutHandler. Supposedly a copy-paste side-effect.
[p171] Check out the first paragraph:

WebSocket is an advanced network protocol that has been developed to improve the performance and responsiveness of web applications. We’ll explore Netty’s support for each of them by writing a sample application.

Each of them? Who are they?
[p177] The call to retain() is needed because after channelRead() … → The call to retain() is needed because after channelRead0() …
[p178, Table 12.1] Identical to Table 11.3.
[p181, Figure 12.3] ChunkedWriteHandler is missing.
[p183, Listing 12.4] There the shutdown of the chat server is realized via Runtime.getRuntime().addShutdownHook(). Is this a recommended practice?
[p189] Figure 14.1 presents a high-level view of the … → Figure 13.1
[p189] Listing 14.1 shows the details of this simple POJO. → Listing 13.1
[p190, Listing 13.1] received field is not used at all. Could be removed to increase clarity. Interestingly, the field is not even encoded.
[p191, Table 13.1] extendsDefaultAddressedEnvelope → extends DefaultAddressedEnvelope
[p191] Figure 14.2 shows the broadcasting of three log … → Figure 13.2
[p192] Figure 14.3 represents a high-level view of the … → Figure 13.3
[p192, Listing 13.2] A byte[] file and byte[] msg pair is encoded as follows:
```
buf.writeBytes(file);
buf.writeBytes(LogEvent.SEPARATOR);
buf.writeBytes(msg);
```
Later on each entry is read back by splitting at LogEvent.SEPARATOR. What if file contains LogEvent.SEPARATOR? I think this is a bad encoding practice. I would rather do:
```
buf.writeInt(file.length);
buf.writeBytes(file);
buf.writeInt(msg.length);
buf.writeBytes(msg);
```
[p194, Listing 13.3] Is there a constant for 255.255.255.255 broadcast address?
[p195] Figure 14.4 depicts the ChannelPipeline of the LogEventonitor … → Figure 13.4
[p196] Check this out:
The LogEventHandler prints the LogEvents in an easy-to-read format that consists of the following:
- The received timestamp in milliseconds.
Really? I did not know epoch timestamps were easy-to-read. Maybe for some definition of easy-to-read.
[p195] Now we need to install our handlers in the ChannelPipeline, as seen in figure 14.4. → Figure 13.4
[p205] Approach A, optimistic and apparently simpler (figure 15.1) → figure 14.1
[p206] Half of the page is spent for justifying Droplr’s preference of approach B (safe and complex) over approach A (optimistic and simpler). Call me an idiot, but I am not sold to these arguments that the former approach is less safe.
[p207] Type of pipelineFactory is missing.
[p210] There is a bullet for tuning JVM. This on its own could have been a really interesting chapter of this book.
[p213] Firebase is indeed implementing TCP-over-long-polling. I wonder if there exists any Java libraries that implements user-level TCP over a certain channel abstraction.
[p214] Figure 15.4 demonstrates how the Firebase long-polling … → Figure 14.4
[p215] Figure 15.5 illustrates how Netty lets Firebase respond to … → Figure 14.5
[p216] … can start as soon as byes come in off the wire. → bytes

[p217, Listing 14.3] Last parenthesis is missing:

rxBytes += buf.readableBytes(
                          tryFlush(ctx)

[p217, Listing 14.3] 70% of the intro was about implementing a control flow over long polling, though the shared code snippet is about totally something else and almost irrelevant.
[p223] In referring to figure 15.1, note that two paths … → figure 14.6
[p229] This request/execution flow is shown in figure 16.1. → figure 15.1
[p230] Figure 16.2 shows how pipelined requests are handled … → Figure 15.2
[p230] …, in the required order. See figure 16.3. → figure 15.3
[p232] That simple flow (show in figure 16.4) works… → figure 15.4
[p232] The client call is dispatched to the Swift library, … What is Swift library? Was not explained anywhere.
[p232] This is the flow shown in figure 16.5. → figure 15.5
[p234] This is a really interesting piece:

Before Nifty, many of our major Java services at Facebook used an older, custom NIO-based Thrift server implementation that works similarly to Nifty. That implementation is an older codebase that had more time to mature, but because its asynchronous I/O handling code was built from scratch, and because Nifty is built on the solid foundation of Netty’s asynchronous I/O framework, it has had many fewer problems.

One of our custom message queuing services had been built using the older framework, and it started to suffer from a kind of socket leak. A lot of connections were sitting around in CLOSE_WAIT state, meaning the server had received a notification that the client had closed the socket, but the server never reciprocated by making its own call to close the socket. This left the sockets in a kind of CLOSE_WAIT limbo.

The problem happened very slowly; across the entire pool of machines handling this service, there might be millions of requests per second, but usually only one socket on one server would enter this state in an hour. It wasn’t an urgent issue because it took a long time before a server needed a restart at that rate, but it also complicated tracking down the cause. Extensive digging through the code didn’t help much either: initially several places looked suspicious, but everything ultimately checked out and we didn’t locate the problem.
[p238] Figure 16.6 shows the relationship between … → figure 15.6
[p239, Listing 15.2] All presented Scala code in this chapter is over-complicated and the complexity does not serve any purpose except wasting space and increasing cognitive load. For instance, why does ChannelConnector extend (SocketAddress => Future[Transport[In, Out]]) rather than just being a simple method?
[p239] This factory is provided a ChannelPipelineFactory, which is … What is this factory?

Conclusion

In summary, Netty in Action is a book that I would recommend to everyone who wants to learn more about Netty to use it in their applications. Almost the entire set of fundamental Netty abstractions are covered in detail. The content is a bliss for novice users in networking domain. Though this in return might make the book uninteresting for people who already got their hands pretty dirty with networking facilities available in Java Platform. That being said, the presented historical perspective and shared case studies are still pretty attractive even for the most advanced users.

I don’t know much about the 2^nd author of the book, Marvin Allen Wolfthal. Though, the 1^st author, Norman Maurer, is a pretty known figure in the F/OSS ecosystem. If he manages to transfer more juice from his experience and presentations to the book, I will definitely buy the 2^nd print of the book too!

Inter-Microservice Communication Fatigue

2017-04-18T17:22:00Z

Let me get this straight: Every single line of code that needs to communicate with a remote microservice is the most bizarre, annoying, sad, and hopeless experience in my daily coding routine. And the worst is: Most of the time its my client code communicating with my services, so there is no one else to blame that would sooth my anger. But I did not end up here out of blue.

In my freshman years, I was given the responsibility of further development of a microservice, where both the server and its driver (API models, HTTP client, etc.) were written in Scala. Because Scala was still a cool buzzword back then and the team wanted to experiment with it. (No, this won’t be a Scala FUD post.) It was using an in-house built HTTP client, which is more or less yet another buggy wrapper over an ancient version of Ning async-http-client. I implemented a (yes, another!) thin wrapper over it to expose the HTTP response models as scala.concurrent.Futures, so we can compose them via Scala’s for-comprehensions. (It did not take long for me to figure out that exposing the API in Scala was one of the worst possible design decisions one could have made in an ecosystem dominated by Java consumers.)

Later on as a team we adopted another critical microservice comprising a giant Java client with Spring fizz and buzz, caching, Guava immutables all under the hood with insanely strict checkNotNull/checkArgument-powered model validation, etc. This comet created its own fan clubs. There are two types of people in the company who are consuming this service:

ones that bite the bullet and use the gigantic driver we provide (say hello to a truck load of artifacts in your not-sufficiently-sucking dependency hell) or
ones that prefer to implement his/her own HTTP driver hosting an empire of bugs by manually building/parsing query request/response models formatted in JSON/XML/Protobuf.

Later on I said enough is enough! Let’s stick to a standard: JEE HTTP client, that is, Jersey JAX-RS Client with Jackson cream on it. I still needed to create all the API models and verify them myself every time. It was bearable to some extent. But here comes the perfect storm: JAX-RS Client 2.0 (which supports proper HTTP connection pooling with sanely configurable socket+connection timeout support, which weren’t sanely available in 1.x) requires javax.ws.rs-api 2.x, which is binary incompatible with 1.x, which is used by 80% of microservices in the ecosystem. So in practice no other microservice will be able to use my driver without the developer losing half of his/her hairs.

Later on I kept repeating “enough is enough”! Let’s use Google’s async-http-client. It is pluggable all around the place: the HTTP connector (Apache HC, etc.), marshaller (Jackson, Gson, etc.). The project is more or less undocumented. But thanks to an army of Android users, there is plenty of blog posts and source code to discover your own personal bugs, so you can keep on blog posting about it. Anyway… It worked. I still need to motivate myself to dive into the source code to comprehend how it works under the hood, but it worked.

Today… When I need to talk to one of these services I need to pick a lousy, juice, smelly sh*t of my preference:

Inject the entire Scala milky way into your 1-file Java microservice, which could have been delivered as a 5MB fat-JAR before Scala space-warping it into 50MB. And don’t forget to pat your IDE in the back every time it needs to auto-complete a Scala class. Oh, by the way, have you ever tried accessing a scala.Option from Java? Boy! It is fun! I hope your service consumers think likewise.
Let the giant Java driver bring all its feature-rich functionality together with its cousins, its nephews, its uncle, its grandma, its grandpa, its friends from the school, its ex, and of course with spring-core. All you wanted is to make a GET to /v1/user/<id>, but now you have the entire Pivotal art gallery decorating your mvn dependency:tree output on the wall.
You can of course purpose maven-shade-plugin to shade and relocate the entire javax.ws.rs-api, Jersey dependencies, together with the entire universe. I know you can do that.
Browse to Google’s async-http-client webpage and try to find the page that explains how to make a simple fscking GET request.
Embrace the old Ning client wrapper, welcome bugs (the first time I needed to use it I found out that setHeaders() wasn’t working as expected), stick to JAX-RS 1.x and an ancient Netty version, which causes a JAR Hell with any recent library, e.g., Elasticsearch. (Please refer back to maven-shade-plugin item.)

I can hear you shouting about compile-time generated HTTP clients based on Swagger or WADL specs. But weren’t we just cursing WSDL and trying to run away from it? Retrofit? Finagle? gRPC? I bet it is a matter of time until you end up needing to consume two clients which have transitive dependencies to two binary incompatible versions of Retrofit/Finagle/gRPC. You can blame the Java class loader mechanism. But that doesn’t make the problem fade away. Oh! I was just about to forget! Wait until I migrate to rx.Completable from rx.Single<Void>, which I migrated from rx.Observable<Void>.

I am exhausted and demotiviated to write yet another single line of code that needs to communicate with a remote microservice and which could have been a simple fucking RPC. I don’t have a solution for the mud ball in my hands. Even if I do have, I am not sure whether it will survive a couple of years or not. But in the back of my head, I keep on cursing the Java Platform SE guys: How difficult could it be to come up with a proper pluggable HTTP client? Compared to NPE, Java’s HTTP client is not the billion dollar mistake, but a really close one.

Guice Integration in Hazelcast

2017-04-18T17:22:00Z

For many occassions I find the distributed ExecutorService of Hazelcast (aka. IExecutorService) pretty convenient to turn a set of nodes into a tamed cluster waiting for orders. You just submit an either Runnable or Callable<T> and Hazelcast takes care of the rest – executing the task on remote members, acknowledging the response(s) back, etc. Though note that since the method and its response will be delivered over the wire, it is no surprise that they all need to be Serializable.

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;
import com.hazelcast.core.Member;
import com.hazelcast.core.MultiExecutionCallback;

import java.io.Serializable;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;

public enum HzGuiceDemo {;

    public static class ProcessorCountTask implements Serializable, Callable<Integer> {

        @Override
        public Integer call() {
            return Runtime.getRuntime().availableProcessors();
        }

    }

    public static void main(String[] args) throws Throwable {
        HazelcastInstance hzInstance = Hazelcast.newHazelcastInstance();
        IExecutorService hzExecutorService = hzInstance.getExecutorService("ballpark");
        CompletableFuture<Integer> totalProcessorCountFuture = new CompletableFuture<>();
        hzExecutorService.submitToAllMembers(
                new ProcessorCountTask(),
                new MultiExecutionCallback() {

                    @Override
                    public void onResponse(Member member, Object value) {
                        // Ignored.
                    }

                    @Override
                    public void onComplete(Map<Member, Object> values) {
                        int totalProcessorCount = values
                            .values()
                            .stream()
                            .mapToInt(object -> (int) object)
                            .sum();
                        totalProcessorCountFuture.complete(totalProcessorCount);
                    }

                });
        int totalProcessorCount = totalProcessorCountFuture.get(10, TimeUnit.SECONDS);
        System.out.format("there are %d processors in total%n", totalProcessorCount);
        hzInstance.shutdown();
    }

}

Unfortunately many of our tasks are not isolated from the rest of the application state (i.e., stateless) as ProcessorCountTask given above. Most of the time the functional requirements necessitate access to the remote node state that is available through beans provided by the underlying dependency injection framework. Consider the following stateful PizzaService that is responsible for cooking pizzas to its users.

import javax.inject.Singleton;

import static com.google.common.base.Preconditions.checkArgument;

@Singleton
public static class PizzaService {

    private volatile int totalPizzaCount = 0;

    public synchronized int cook(int amount) {
        checkArgument(amount > 0, "expecting: amount > 0, found: %s", amount);
        availablePizzaCount += amount;
        System.out.format("🍕 cooking %d pizza(s)%n", amount);
    }

}

We further have a task class to remotely command PizzaService to cook:

import java.io.Serializable;

public static class PizzaCookTask implements Serializable, Runnable {

    @Inject
    private PizzaService pizzaService;

    private final int amount;

    public PizzaMakeTask(int amount) {
        this.amount = amount;
    }

    @Override
    public void run() {
        pizzaService.cook(amount);
    }

}

A naive approach to run this task on an IExecutorService would result in the following code:

import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IExecutorService;

import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;

public enum HzGuiceDemo {;

    public static void main(String[] args) throws Throwable {
        HazelcastInstance hzInstance = Hazelcast.newHazelcastInstance();
        IExecutorService hzExecutorService = hzInstance.getExecutorService("ballpark");
        hzExecutorService.executeOnAllMembers(new PizzaCookTask(1));
        hzInstance.shutdown();
    }

}

which fails with a sweet NullPointerException as follows:

Exception in thread "main" java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
  at com.vlkan.hzguicedemo.HzGuiceDemo.main(HzGuiceDemo.java:??)
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at com.hazelcast.executor.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:189)
  at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:186)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
  at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
Caused by: java.lang.NullPointerException
  at com.vlkan.hzguicedemo.HzGuiceDemo$PizzaCookTask.call(HzGuiceDemo.java:??)
  at com.vlkan.hzguicedemo.HzGuiceDemo$PizzaCookTask.call(HzGuiceDemo.java:??)

What is really happening here is that Hazelcast does not have a magical ball to guess the dependency injection framework you are using to process the @Inject-annotated properties of the PizzaCookTask. Though Hazelcast has something else: ManagedContext. In a nutshell, ManagedContext provides means to intercept class instantiation at deserialization. We can leverage this functionality to come up with a ManagedContext implementation that bakes Guice dependency injection into the Hazelcast class instantiation process.

import com.google.inject.Injector;
import com.hazelcast.core.ManagedContext;

import javax.inject.Inject;
import javax.inject.Singleton;

@Singleton
public class HazelcastGuiceManagedContext implements ManagedContext {

    private final Injector injector;

    @Inject
    public HazelcastGuiceManagedContext(Injector injector) {
        this.injector = injector;
    }

    @Override
    public Object initialize(Object instance) {
        injector.injectMembers(instance);
        return instance;
    }

}

Next all you need to do is to use this ManagedContext while creating your HazelcastInstance:

Injector injector = Guice.createInjector();
HazelcastGuiceManagedContext guiceManagedContext = injector.getInstance(HazelcastGuiceManagedContext.class);
Config hzConfig = new Config();
hzConfig.setManagedContext(guiceManagedContext);
HazelcastInstance hzInstance = Hazelcast.newHazelcastInstance(hzConfig);

While I have provided an example for Guice, this method is applicable to any dependency injection framework that provides an equivalent to Injector#injectMembers() of Guice. Needless to say, but Spring folks are already covered by SpringManagedContext shipped with Hazelcast.

Notes on "Coders at Work"

2016-10-04T18:40:00Z

There is nothing like thinking about work while your are on vacation. And that was indeed what I did: Reading Coders at Work: Reflections on the Craft of Programming in a tango-themed Buenos Aires trip.

I had already met with Peter Seibel in his well-known splendid work: Practical Common Lisp. He definitely has a knack for transforming technically challenging problems into a pedagogically digestable nootropic. This uncanny ability was placidly lurid in Coders at Work as well. Motivated by the mind provoking exquisite content, I felt an urge to keep a record of its reflections on me.

On the Content

I totally enjoyed the book and read it cover to cover. Nevertheless, I believe the following subtleties could have been addressed in a better way.

A majority of the interviewed coders are not actively coding any more. I find this a little bit contradictory with the title of the book. While the content still makes a great deal about the historical progress of programming and programmers, I find the detachment of the interviewees from the modern computing slightly unexpected.
Given the back that the book examines the events dating back to more than half a century, I sometimes find myself lost in the time context. Additional footnotes to enhance these kind of ambiguities could have been useful.

Highligts

Below I collected my personal highligts on certain statements that are shared by certain interviewees.

In general, coders do not practice software testing extensively. Further, I had the impression that that they do not read much programming books either. This issue sometimes acclaimed to the lack of necessary set of fundamental books at the early stages of their career.
Among the entire deck, I find Joshua Bloch, Bernie Cosell, and Donald Knuth the ones with the most sensible and to-the-earth statements.
A notable subset of the interviewees dragged into computers not by a deliberate decision, but by chosing a yet another career path that was available to them. (For instance, Fran Allen got a Fortran instructor position in IBM in order to finance her school loans to be able to continue her math teacher career pursuit.)
None believes that reading Knuth’s The Art of Computer Programming is a must read for programmers, nevertheless they acknowledge that it is good to have it under your hand for reference.
Except Knuth himself, nobody practices literate programming. (I am astounded to observe how Seibel is biased to ask this non-sense question which delivers no practical value at all to every single candidate.)
Majority agrees that programming is a way more complicated and challenging occupation than it once used to be in the past.
More than half thinks that good writing skills are a big plus (some even state necessity) for programming.
printf is the clear winner as the debugging tool of preference among interviewees.

Quotes

Below you can find some snippets that I find worth mentioning from the book.

Jamie Zawinski

I wish I would have known this when I was in high school. Could not agree more.

Zawinski: When you’re in high school, everyone tells you, “There’s a lot of repetitive bullshit and standardized tests; it’ll all be better once you’re in college.” And then you get to your first year of college and they’re like, “Oh, no – it gets better when you’re in grad school.” So it’s just same shit, different day – I couldn’t take it. [p5]

His comments on C++, which are shared by many other interviewees throughout the book:

Zawinski: … when you’re programming C++ no one can ever agree on which ten percent of the language is safe to use. [p20]

The sad truth about F/OSS:

Seibel: Isn’t it exactly this thing – someone comes along and says, “I can’t understand this stuff. I’ll just rewrite it” – that leads to the endless rewriting you bemoan in open-source development?

Zawinski: Yeah. But there’s also another aspect of that which is, efficiency aside, it’s just more fun to write your own code than to figure out someone else’s. So it is easy to understand why that happens. But the whole Linux/GNOME side of things is straddling this line between someone’s hobby and a product. Is this a research project where we’re deciding what desktops should look like and we’re experimenting? Or are we competing with Macintosh? Which is it? Hard to do both. [p23]

Brad Fitzpatrick

His thoughts on finishing a project, which I sadly share as well:

Fitzpatrick: The projects that I never finish … it’s because I did the hard part and I learned what I wanted to learn and I never got around to doing the boring stuff. [p20]

He is also poisoned by LWN, Reddit, etc.

Fitzpatrick: I like working alone but I just bounce all over the place when I do. On a plane I’ll bring extra laptop batteries and I have a whole development environment with local web servers and I’ll be in a web browser, testing stuff. But I’ll still be hitting new tabs, and typing “reddit” or “lwn” – sites I read. Autocomplete and hit Enter, and then – error message. I’ll do this multiple times within a minute. Holy fuck! Do I do this at work? Am I reading web site this often that I don’t even think about it? It’s scary. I had a friend, who had some iptables rule, that on connection to a certain IP address between certain hours of the day would redirect to a “You should be working” page. I haven’t got around to doing that, but I need to do something like it, probably. [p73]

Douglas Crockford

Why programming is difficult?

Crockford: Part of what makes programming difficult is most of the time we’re doing stuff we’ve never done before. [p110]

He talks about his preferred way for interviewing job candidates, which is also shared by other coders in the book.

Crockford: The approach I’ve taken now is to do a code reading. I invite the candidate to bring in a piece of code he’s really proud of and walk us through it. [p129]

Brendan Eich

Nothing noteworthy, you may guess why.

Joshua Bloch

Is Java off in the weeds?

Seibel: … is Java off in the weeds a little bit? Is it getting more complex faster than it’s getting better?

Bloch: That’s a very difficult question. In particular, the Java 5 changes added far more complexity than we ever intended. I had no understanding of just how much complexity generics and, in particular, wildcards were going to add to the language. I have to give credit where is due – Graham Hamilton did understand this at the time and I didn’t.

The funny things is, he fought against it for years, trying to keep generics out of the language. But the notion of variance – the idea behind wildcards – came into fashion during the years when generics were successfully being kept out of Java. If they had gone in earlier, without variance, we might have had a simpler, more tractable language today.

That said, there are real benefits to wildcards. There’s a fundamental impedance mismatch between subtyping and generics, and wildcards go a long way towards rectifying the mismatch. But at a significant cost in terms of complexity. THere are some people who believe that declaration-site, as opposed to use-site, variance is a better solution, but I’m not so sure.

The jury is basically still out on anything that hasn’t been tested by a huge quantity of programmers under real-world conditions. Often languages only succeed in some niche and people say, “Oh, they’re great and it’s such a pity they didn’t become the successful language in the world.” But often there are reasons they didn’t. Hopefully some language that does use declaration-site variance, like Scala or C# 4.0, will answer this question once and for all. [p191]

On “obviously no deficiencies” versus “no obvious deficiencies”:

Bloch: There’s a brilliant quote by Tony Hoare in his Turing Award speech about how there are two ways to design a system: “One way is to make it so simple that there are obviously no deficiencies and the other way is to make it is so complicated that there are no obvious deficiencies.”

The paragraph that follows is equally brilliant, though it isn’t as well-known: “The first method is far more difficult. It demands the same skill, devotion, insight, and even inspiration as the discovery of the simple physical laws which underlie the complex phenomena of nature. It also requires a willingness to accept objectives which are limited by physical, logical, and technological constraints, and to accept a compromise when conflicting objectives cannot be met. No committee will ever do this until it is too late.” [p197]

Smart people and programming:

Seibel: Speaking of writing intricate code, I’ve noticed that people who are too smart, in a certain dimension anyway, make the worst code. Because they can actually fit the whole thing in their head they can write these great reams of spaghetti code.

Bloch: I agree with you that people who are both smart enough to cope with enormous complexity and lack empathy with the rest of use may fail prey to that. They think, “I can understand this and I can use it, so it has to be good.” [p202]

…

There’s this problem, which is, programming is so much of an intellectual meritocracy and often these people are the smartest people in the organization; therefore they figure they should be allowed to make all the decisions. But merely the fact that they’re the smartest people in the organization doesn’t mean they should be making all the decisions, because intelligence is not a scalar quantity; it’s a vector quantity. And if you lack empathy or emotional intelligence, then you shouldn’t be designing APIs or GUIs or languages. [p203]

Joe Armstrong

On paralyses of choice:

Armstrong: The funny thing is, thinking back, I don’t think all these modern gizmos actually make you any more productive. Hierarchical file systems – how do they make you more productive? Most of software development goes on in your head anyway. I think having worked with that simpler system imposes a kind of disciplined way of thinking. If you haven’t got a directory system and you have to put all the files in one directory, you have to be fairly disciplined. If you haven’t got a revision control system, you have to be fairly disciplined. Given that you apply that discipline to what you’re doing it doesn’t seem to me to be any better to have hierarchical file systems and revision control. They don’t solve the fundamental problem of solving your problem. They probably make it easier for groups of people to work together. For individuals I don’t see any difference.

Also, I think today we’re kind of overburdened by choice. I mean, I just had Fortran. I don’t think we even had shell scripts. We just had batch files so you could run things, a compiler, and Fortran. And assembler possibly, if you really needed it. So there wasn’t this agony of choice. Being a young programmer today must be awful – you can choose 20 different programming languages, dozens of framework and operating systemsand you’re paralyzed by choice. There was no paralysis of choice then. You just start doing it because the decision as to which language and things is just made – there’s no thinking about what you should do, you just go and do it. [p210]

Simon Peyton Jones

Testing an API in Microsoft:

Peyton Jones: Well, they also do some interesting work on testing APIs. Steven Clarke and his colleagues at Redmond have made systematic attempts to watch programmers, given a new API, talk through what they’re trying to do. And they get the people who designed the API to sit behind a glass screen and watch them.

And the guys sitting there behind the glass screen say, “No, no, don’t do that! That’s not the right way!” But it’s soundproof. That turns out often to be very instructive. They go and change their API. [p253]

Peter Norvig

On the traditional master and apprentice approach:

Norvig: But I think part of the reasons why you had master and apprentice is because the materials were rarer. When you were doing goldsmithing, there’s only so much gold. Or when the surgeon’s operating, there’s only one heart, and so you want the best person on that and you want the other guys just helping. With coding, it’s not like that. You’ve got plenty of terminals. You’ve got plenty of keyboards. You don’t have to ration it. [p295]

Why programming is not an art, but a craft:

Seibel: As a programmer, do you consider yourself a scientist, an engineer, an artist, or a craftsman?

Norvig: Well, I know when you compare the various titles of books and so on, I always thought the “craft” was the right answer. So I thought art was a little pretentious because the purpose of art is to be beautiful or to have an emotional contact or emotional impact, and I don’t feel like that’s anything that I try to do. Certainly I want programs to be pretty in some ways, and sometimes I feel like I spend too much time doing that. I’ve been in a position where I’ve had the luxury to say, “Gee, I have time to go back and pretty this up a little bit.” And places where I’ve been able to write for a publication, you spend more time doing that than you would if it was just for your own professional growth.

But I don’t think of that as art. I think craft is really the right word for it. You can make a chair, and it’s good looking, but it’s mostly functional – it’s a chair. [p319]

Guy Steele

On the difficulty of getting a program right:

Steele: I’ll give you another example – suppose I were to tell my smart computer, “OK, I’ve got this address book and I want the addresses to always be in sorted order,” and it responds by throwing away everything but the first entry. Now the address book is sorted. But that’s not what you wanted. It turns out that just specifying something as simple as “a list is in sorted order and I haven’t lost any of the data and nothing has been duplicated” is actually a fairly tricky specification to write. [p361]

Dan Ingalls

Was a nice read, though I could not find anything particularly interesting worth sharing. Nevertheless, along the lines Seibel says something that I have never heard of:

Seibel: Alan Kay has said that both Lisp and Smalltalk have the problem that they’re so good they eat their children. If you had known Lisp, then Smalltalk would have been the first eaten child. [p378]

L Peter Deutsch

On getting data structures right:

Deutsch: … if you get the data structures and their invariants right, most of the code will just kind of write itself. [p420]

Conceptualization of software and memory pointers:

Deutsch: … I don’t look around and see anything that looks like an address or a pointer. We have objects; we don’t have these weird things that computer scientists misname “objects.”

Seibel: To say nothing of the scale. Two to the 64th of anything is a lot, and things happening billions of times a second is fast.

Deutsch: But that doesn’t bother us here in the real world. You know Avogadro’s number, right? Ten to the 23rd? So, we’re looking here around at a world that has incredible numbers of little things all clumped together and happening at the same time. It doesn’t bother us because the world is such that you don’t have to understand this table at a subatomic level. The physical properties of matter are such that 99.9 percent of the time you can understand it in aggregate. And everything you have to know about it, you can understand from dealing with it in aggregate. To a great extent, that is not true in the world of software.

People keep trying to do modularization structures for software. And the state of that art has been improving over time, but it’s still, in my opinion, very far away from the ease with which we look around and see things that have, whatever it is, 10 to the 23rd atoms in them, and it doesn’t even faze us.

Software is a discipline of detail, and that is a deep, horrendous fundamental problem with software. Until we understand how to conceptualize and organize software in a way that we don’t have to think about how every little piece interacts with every other piece, things are not going to get a whole lot better. And we’re very far from being there. [p424]

Ken Thompson

On teaching:

Thompson: … I love the teaching: the hard work of a first class, the fun of the second class. Then the misery of the third. [p455]

What I am supposed to do and what I am actually doing:

Thompson: We were supposed to be doing basic research but there was some basic research we should be doing and some basic research we shouldn’t be doing. And just coming out of the ashes of MULTICS, operating systems was one of those basic research things we shouldn’t be doing. Because we tried it, it didn’t work, it was a huge failure, it was expensive; let’s drop it. So I kind of expected that for what I was doing I was going to eventually get fired. I didn’t. [p458]

Code rots:

Thompson: Code by itself almost rots and it’s gotta be rewritten. Even when nothing has changed, for some reason it rots. [p460]

10 percent of the work:

Thompson: NELIAC was a system-programming version of Algol 58.

Seibel: Was Bliss also from that era?

Thompson: Bliss I think was after. And their emphasis was trying to compile well. I think it was pretty clear from the beginning that you shouldn’t kill yourself compiling well. You should do well but not really good. And the reason is that in the time it takes you to go from well to really good, Moore’s law has already surpassed you. You can pick up 10 percent but while you’re picking up that 10 percent, computers have gotten twice as fast and maybe with some other stuff that matters more for optimization, like caches. I think it’s largely a waste of time to do really well. It’s really hard; you generate as many bugs as you fix. You should stop, not take that extra 100 percent of time to do 10 percent of the work. [p462]

Writing an OS to test a file system:

Seibel: So you basically wrote an OS so you’d have a better environment to test your file system.

Thompson: Yes. Halfway through there that I realized it was a real time- sharing system. I was writing the shell to drive the file system. And then I was writing a couple other programs that drove the file system. And right about there I said, “All I need is an editor and I’ve got an operating system.” [p465]

Economics of deciding on introducing a bag:

Thompson: Certainly every time I’ve written one of these non-compare subroutine calls, strcpy and stuff like that, I know that I’m writing a bug. And I somehow take the economic decision of whether the bug is worth the extra arguments. [p468]

On testing:

Thompson: … Mostly just regression tests.

Seibel: By things that are harder to test, you mean things like device drivers or networking protocols?

Thompson: Well, they’re run all the time when you’re actually running an operating system.

Seibel: So you figure you’ll shake the bugs out that way?

Thompson: Oh, absolutely. I mean, what’s better as a test of an operating system than people beating on it? [p469]

Code at Google:

Thompson: I guess way more than 50 percent of the code is the what-if kind. [p473]

On literate programming:

Seibel: When I interviewed him, Knuth said the key to technical writing is to say everything twice in complementary ways. So I think he sees that as a feature of literate programming, not a bug.

Thompson: Well if you have two ways, one of them is real: what the machine executes. [p477]

Fran Allen

What makes a program beautiful?

Allen: That it is a simple straightforward solution to a problem; that has some intrinsic structure and obviousness about it that isn’t obvious from the problem itself. [p489]

Bernie Cosell

Should we teach Knuth to students?

Cosell: I would not teach students Knuth per se for two reasons. First, it’s got all this mathematical stuff where he’s not just trying to present the algorithms but to derive whether they’re good or bad. I’m not sure you need that. I understand a little bit of it and I’m not sure I need any of it. But getting a feel for what’s fast and what’s slow and when, that’s an important thing to do even if you don’t know how much faster or how much slower.

The second problem is once students get sensitive to that, they get too clever by half. They start optimizing little parts of the program because, “This is the ideal place to do an AB unbalanced 2-3 double reverse backward pointer cube thing and I always wanted to write one of those.” So they spend a week or two tuning an obscure part of a program that doesn’t need anything, which is now more complicated and didn’t make the program any better. So they need a tempered understanding that there are all these algorithms, how they work, and how to apply them. It’s really more of a case of how to pick the right one for the job you’re trying to do as opposed to knowing that this one is an order n-cubed plus three and this one is just order n-squared times four. [p527]

Writing programs and learning how to program:

Cosell: The binary bits are what computers want and the text file is for me. I would get people – bright, really good people, right out of college, tops of their classes – on one of my projects. And they would know all about programming and I would give them some piece of the project to work on. And we would start crossing swords at our project-review meetings. They would say, “Why are you complaining about the fact that I have my global variables here, that I’m not doing this, that you don’t like the way the subroutines are laid out? The program works.”

They’d be stunned when I tell them, “I don’t care that the program works. The fact that you’re working here at all means that I expect you to be able to write programs that work. Writing programs that work is a skilled craft and you’re good at it. Now, you have to learn how to program.” [p543]

Convictions:

Cosell: I had two convictions, which actually served me well: that programs ought to make sense and there are very, very few inherently hard problems. [p549]

How long is it going to take you to put this change in?

Cosell: So when they ask, “How long is it going to take you to put this change in?” you have three answers. The first is the absolute shortest way, changing the one line of code. The second answer is how long it would be using my simple rule of rewriting the subroutine as if you were not going to make that mistake. Then the third answer is how long if you fix that bug if you were actually writing this subroutine in the better version of the program. [p550]

Artistry in programming:

Cosell: Part of what I call the artistry of the computer program is how easy it is for future people to be able to change it without breaking it. [p555]

Difficulty of programming and C:

Cosell: … programmers just can’t be careful enough. They don’t see all the places. And C makes too many places. Too scary for me, and I guess it’s fair to say I’ve programmed C only about five years less than Ken has. We’re not in the same league, but I have a long track record with C and know how difficult it is and I think C is a big part of the problem. [p559]

75 million run-of-the-mill programmers and Java:

Cosell: When I first messed with Java – this was when it was little baby language, of course – I said, “Oh, this is just another one of those languages to help not-so-good programmers go down the straight and narrow by restricting what they can do.” But maybe we’ve come to a point where that’s the right thing. Maybe the world has gotten so dangerous you can’t have a good, flexible language that one percent or two percent of the programmers will use to make great art because the world is now populated with 75 million run-of-the-mill programmers building these incredibly complicated applications and they need more help than that. So maybe Java’s the right thing. I don’t know. [p560]

Not-so-good programmers and C:

Cosell: I don’t want to say that C has outlived its usefulness, but I think it was used by too many good programmers so that now not-good-enough programmers are using it to build applications and the bottom line is they’re not good enough and they can’t. Maybe C is the perfect language for really good systems programmers, but unfortunately not-so-good systems and applications programmers are using it and they shouldn’t be. [p560]

Donald Knuth

Teaching a class, writing a book, and programming:

Knuth: I could teach classes full-time and write a book full-time but software required so much attention to detail. It filled that much of my brain to the exclusion of other stuff. So it gave me a special admiration for people who do large software projects – I would never have guessed it without having been faced with that myself. [p572]

Why isn’t everybody a super programmer and super writer?

Knuth: Now, why hasn’t this spread over the whole world and why isn’t everybody doing it? I’m not sure who it was who hit the nail on the head – I think it was Jon Bentley. Simplified it is like this: only two percent of the world’s population is born to be super programmers. And only two percent of the population is born to be super writers. And Knuth is expecting everybody to be both. [p574]

Use of pointers in C:

Knuth: To me one of the most important revolutions in programming languages was the use of pointers in the C language. When you have nontrivial data structures, you often need one part of the structure to point to another part, and people played around with different ways to put that into a higher- level language. Tony Hoare, for example, had a pretty nice clean system but the thing that the C language added – which at first I thought was a big mistake and then it turned out I loved it – was that when x is a pointer and then you say, x + 1 , that doesn’t mean one more byte after x but it means one more node after x , depending on what x points to: if it points to a big node, x + 1 jumps by a large amount; if x points to a small thing, x + 1 just moves a little. That, to me, is one of the most amazing improvements in notation. [p585]

I did not know about Knuth’s change files. But it seemed like an inconvenient overkill:

Knuth: I had written TeX and Metafont and people started asking for it. And they had 200 or 300 combinations of programming language and operating system and computer, so I wanted to make it easy to adapt my code to anybody’s system. So we came up with the solution that I would write a master program that worked at Stanford and then there was this add-on called a change file which could customize it to anybody else’s machine.

A change file is a very simple thing. It consists of a bunch of little blobs of changes. Each change starts out with a few lines of code. You match until you find the first line in the master file that agrees with the first line of your change. When you get to the end of the part of the change that was supposed to match the master file, then comes the part which says, “Replace that by these lines instead.” [p586]

The extreme example of this was when TeX was adapted to Unicode. They had a change file maybe 10 times as long as the master program. In other words, they changed from an 8-bit program to a 16-bit program but instead of going through and redoing my master program, they were so into change files that they just wrote their whole draft of what they called Omega as change files, as a million lines of change files to TeX’s 20,000 lines of code or something. So that’s the extreme. [p587]

Is programming fun any more?

Knuth: So there’s that change and then there’s the change that I’m really worried about: that the way a lot of programming goes today isn’t any fun because it’s just plugging in magic incantations – combine somebody else’s software and start it up. It doesn’t have much creativity. I’m worried that it’s becoming too boring because you don’t have a chance to do anything much new. [p594]

Code reading:

Knuth: … don’t only read the people who code like you. [p601]

Programmatically Taking Heap and Thread Dumps in HotSpot

2016-08-12T17:53:00Z

While taking heap and thread dumps are one click away using modern JVM toolset, in many cases the deployment environment access restrictions render these options unusable. Hence, you might end up exposing these functionalities in certain ways like an internal REST interface. This implies a new nasty obstacle: You need to know how to programmatically take heap and thread dumps in a Java application. Unfortunately, there does not exist a standard interface to access these functionalities within the VM as of date. But if you are only concerned about HotSpot, then you are in luck!

Heap Dumps

For heap dumps, once you get your teeth into a HotSpotDiagnosticMXBean, you are safe to go. It already exposes a dumpHeap() method ready to be used.

import com.sun.management.HotSpotDiagnosticMXBean;

import javax.management.MBeanServer;
import java.io.File;
import java.io.IOException;
import java.lang.management.ManagementFactory;

public enum HotSpotHeapDumps {;

    private static final HotSpotDiagnosticMXBean HOT_SPOT_DIAGNOSTIC_MX_BEAN =
            getHotspotDiagnosticMxBean();

    private static HotSpotDiagnosticMXBean getHotspotDiagnosticMxBean() {
        MBeanServer server = ManagementFactory.getPlatformMBeanServer();
        try {
            return ManagementFactory.newPlatformMXBeanProxy(
                server, HOT_SPOT_DIAGNOSTIC_MX_BEAN_NAME, HotSpotDiagnosticMXBean.class);
        } catch (IOException error) {
            throw new RuntimeException("failed getting Hotspot Diagnostic MX bean", error);
        }
    }

    public void create(File file, boolean live) throws IOException {
        HOT_SPOT_DIAGNOSTIC_MX_BEAN.dumpHeap(file.getAbsolutePath(), live);
    }

}

The second argument of dumpHeap denotes live objects, that is, objects that are reachable from others.

Note that many real-world Java applications occupy quite some memory. As a result of this, created heap dump generally end up consuming significant amount of disk space. You need to come up with your own custom clean up mechanism to tackle this problem. (For instance, in a JAX-RS resource, you can purpose a custom MessageBodyWriter to delete the file after writing the entire file to the output stream.)

Thread Dumps

When you think first about thread dumps, they just contain simple plain text data.

2016-08-12 18:40:46
Full thread dump OpenJDK 64-Bit Server VM (25.76-b198 mixed mode):

"RMI TCP Connection(266)-127.0.0.1" #24884 daemon prio=9 os_prio=0 tid=0x00007f9474010000 nid=0x2cee runnable [0x00007f941571b000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    - locked <0x00000005c086e8b0> (a java.io.BufferedInputStream)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$83/628845041.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00000005c0489198> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JobScheduler FJ pool 0/4" #24883 daemon prio=6 os_prio=0 tid=0x00007f946415d800 nid=0x2ced waiting on condition [0x00007f94093d2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000005d8a5f9e0> (a jsr166e.ForkJoinPool)
    at jsr166e.ForkJoinPool.awaitWork(ForkJoinPool.java:1756)
    at jsr166e.ForkJoinPool.scan(ForkJoinPool.java:1694)
    at jsr166e.ForkJoinPool.runWorker(ForkJoinPool.java:1642)
    at jsr166e.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:108)

   Locked ownable synchronizers:
    - None

Unfortunately, thread dumps do not have a standard syntax. While there are various ways to produce this output, thread dump analysis software does not play well with them. For instance, IBM Thread and Monitor Dump Analyzer for Java cannot parse thread dumps created by VisualVM using JMX. At the end of the day, I always needed to fall back to a HotSpot thread dump.

tools.jar shipped with JDKs (>=1.6) provide the magical HotSpotVirtualMachine class containing our saviour remoteDataDump() method. First add the following lines to your pom.xml:

<dependencyManagement>
    <dependencies>

        <dependency>
            <groupId>com.sun</groupId>
            <artifactId>tools</artifactId>
            <version>${java.version}</version>
            <scope>system</scope>
            <systemPath>${tools.jar}</systemPath>
        </dependency>

    </dependencies>
</dependencyManagement>

<profiles>

    <!-- tools.jar path for GNU/Linux and Windows -->
    <profile>
        <id>default-tools.jar</id>
        <activation>
            <file>
                <exists>${java.home}/../lib/tools.jar</exists>
            </file>
        </activation>
        <properties>
            <tools.jar>${java.home}/../lib/tools.jar</tools.jar>
        </properties>
    </profile>

    <!-- tools.jar path for OSX -->
    <profile>
        <id>default-tools.jar-mac</id>
        <activation>
            <file>
                <exists>${java.home}/../Classes/classes.jar</exists>
            </file>
        </activation>
        <properties>
            <tools.jar>${java.home}/../Classes/classes.jar</tools.jar>
        </properties>
    </profile>

</profiles>

Then the rest is a matter of accessing to HotSpotVirtualMachine class:

import com.google.common.io.ByteStreams;
import com.sun.management.HotSpotDiagnosticMXBean;
import com.sun.tools.attach.AttachNotSupportedException;
import com.sun.tools.attach.VirtualMachine;
import sun.tools.attach.HotSpotVirtualMachine;

import java.io.IOException;
import java.io.InputStream;
import java.lang.management.ManagementFactory;

public enum HotSpotThreadDumps {;

    public String create() throws AttachNotSupportedException, IOException {

        // Get the PID of the current JVM process.
        String selfName = ManagementFactory.getRuntimeMXBean().getName();
        String selfPid = selfName.substring(0, selfName.indexOf('@'));

        // Attach to the VM.
        VirtualMachine vm = VirtualMachine.attach(selfPid);
        HotSpotVirtualMachine hotSpotVm = (HotSpotVirtualMachine) vm;

        // Request a thread dump.
        try (InputStream inputStream = hotSpotVm.remoteDataDump()) {
            byte[] bytes = ByteStreams.toByteArray(inputStream);
            return new String(bytes);
        }

    }

}

You finished writing this code, you clicked on the Run button of the IDE, and it worked like a charm. This get you so excited that you wanted to add this functionality to your JEE service! Or better: Turn this into a JAR and pass it to your client’s machine and watch them take their part in the joy of thread-dump-oriented debugging! And this is what you get in return:

java.lang.NoClassDefFoundError: com/sun/tools/attach/AttachNotSupportedException

Which indicates that you did not pay attention my words: tools.jar is shipped with JDKs. So neither your flashy JEE application server, nor your client’s machine has a JDK, but a JRE. Rings a bell? Yes, you indeed can add tools.jar into the final WAR/JAR of your project:

<build>
    <plugins>

        <!-- copy tools.jar from JAVA_HOME -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-system-dependencies</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/lib</outputDirectory>
                        <includeScope>system</includeScope>
                    </configuration>
                </execution>
            </executions>
        </plugin>

    </plugins>
</build>

Note that this approach incorporates a JDK-specific JAR into your application and assumes that the application will run on a HotSpot VM. But unfortunately this is the only way that I know of to produce a thread dump that works with thread dump analysis software. If you don’t have such a need and just want a crude JMX generated thread dump, check out JmxSupport.java shipped with VisualVM.

Volkan Yazıcı

The State of Lightweight Threads for the JVM

Elasticsearch Survival Guide for Developers

Table of Contents

Mapping

Avoid nested fields

Have a strict mapping

Don’t analyze fields of type string unless necessary

Settings

Avoid oversharding

Unlearn every hack for tuning merges

Pay attention to JVM memory settings

Querying

Compare-and-swap over _version field is poor man’s transactions

Try splitting complex queries

Know your numeric types

Don’t use Elasticsearch Transport/Node client in your Java application (and always use JSON over HTTP)

Use the official Elasticsearch REST client in your Java application

Don’t use Elasticsearch query models to generate cache keys

Don’t use HTTP caching for caching Elasticsearch responses

Use sliced scrolls sorted on _doc

Prefer GET /index/type/{id} over POST /index/_search for single document retrieval

Use size: 0 and includes/excludes wisely

Implement proper backpressure while querying

Provide explicit timeouts in queries

Don’t block the Elasticsearch client I/O threads (and know your threads)

Don’t write Elasticsearch queries with JSON templates injecting variables

Prefer your own JSON serializer over the one provided by Elasticsearch clients

Strategy

Always (try to) stick to the latest JVM and Elasticsearch versions

Use Elasticsearch complete and partial snapshots for backups

Have a continuous performance test bed

Use aliases

Avoid having big synonym collections

Force merge and increase operation bandwidth before enabling replicas

Record application-level metrics

Invest in CPU!

Avoid writing custom Elasticsearch plugins

Notes on "Reactive Programming in Spring 5"

Overview

Pros

Cons

Notes

Using Elasticsearch as the Primary Data Store

Table of Contents

The Search

What is search anyway?

Who/What is using search?

What about performance?

How volatile is the content?

The ETL

Real-time Content Stream

Configuration Stream

Operational Overview

Configuration Mutations

Configuration Predicates

The Old ETL

The Battle of Storage Engines

Benchmark Setup

Benchmark Results

The New ETL

The Primary Storage: Elasticsearch

The Configuration DSL: JSON and Groovy

Conclusion

Acknowledgements

F.A.Q.

Did you try tuning the PostgreSQL optimization knobs?

How do you calculate search_rank?

Have you ever presented this material in a conference setting?

Varnishing Search Performance

Notes on "Netty in Action"

Netty in Action

Things that are well done

Things that could have been improved

Notes

Conclusion

Inter-Microservice Communication Fatigue

Guice Integration in Hazelcast

Notes on "Coders at Work"

On the Content

Avoid `nested` fields

Don’t analyze fields of type `string` unless necessary

Compare-and-swap over `_version` field is poor man’s transactions

Use sliced scrolls sorted on `_doc`

Prefer `GET /index/type/{id}` over `POST /index/_search` for single document retrieval

Use `size: 0` and `includes`/`excludes` wisely

How do you calculate `search_rank`?