How Elastisearch and OpenSearch built their SDKs

Source: Internet Archive

Hi, this is Quentin, I currently work at Elastic, and maintain the Python SDK. Previously, I spent a decade maintaining and building SDKs.

This post is an addition to “Building great SDKs” by me which is published in The Pragmatic Engineer. In the article, I share how Elastisearch and OpenSearch built their respective SDKs. In writing this article, I reached out to Thomas Farr, maintainer of the OpenSearch Java client and API specification. He kindly provided personal feedback on an early draft of this post, which I have partially incorporated. 

Note: the observations and views below are all mine. They are not an official representation of OpenSearch Project/Foundation, Amazon, or Elastic. All errors are also mine.

Elasticsearch and OpenSearch (forked in 2021 from Elasticsearch) each offer an API with hundreds of endpoints, and millions of users have used those APIs! Although now vastly different, they are still similar enough to be interesting to analyze, especially given we use three approaches set out in the The Pragmatic Engineer deepdive:

  • custom generator – the approach used by Elasticsearch
  • general-purpose generator – an approach that OpenSearch experimented with
  • OpenAPI-based generation – currently used by OpenSearch

AWS maintains Smithy and uses it to model various AWS APIs and generate AWS SDKs, and it was the natural choice for a project launched by AWS. OpenSearch tried it but ultimately abandoned it in favor of OpenAPI. Indeed, Smithy enforced too many constraints, and would not express URL ambiguity or untagged unions, whereas OpenAPI is much more permissive. Additionally, OpenAPI is so widely known that adopting it would lower contribution barriers. But how did OpenSearch overcome the limitations of OpenAPI?

  1. While the OpenSearch specification is written manually (unusually for OpenAPI!), it uses many small YAML files, which a preprocessor combines to generate the full specification. It is still verbose OpenAPI 3.x code, but at least it’s maintainable.
  2. Like Elasticsearch’s OpenAPI export, OpenSearch has to duplicate APIs with optional segments, and uses a custom OpenAPI extension (x-operation-group) to clarify which URLs map to the same underlying API.
  3. OpenSearch uses 10 OpenAPI extensions in total! While this is a common strategy to address OpenAPI limitations, other OpenAPI tooling like Swagger ignores those extensions and can’t fully represent it. As a result, OpenSearch documentation is not yet OpenAPI-based.
  4. That said, OpenSearch uses custom linters to validate the specification as well as third-party OpenAPI & JSON Schema tooling, which ensures the specification and its additions are valid.

So, how are OpenSearch SDKs generated? We’ll focus on the OpenSearch Java client, forked from the Elasticsearch Java client in 2021, and initially generated from the Elasticsearch specification. Since the Java generator is not public, OpenSearch initially forked the generated client. Looking at the commit history shows that Thomas Farr is working on generating increasingly large parts of the client from the OpenSearch specification, replacing more and more code inherited from the Elasticsearch Java client, without breaking compatibility. Let’s look at how this was done with two examples.

Enumerations. OpenAPI does not allow describing enum members: it’s a simple list of strings. To describe enums, OpenSearch uses oneOf in some cases. Since enums are not labelled explicitly as such, the Java generator recognizes enums using a complex set of rules.

Shortcut properties, as documented in the specification repository. For example, Elasticsearch and therefore OpenSearch expose a _source field in request bodies, which can be a boolean or a SourceFilter object with “includes” and “excludes” fields. These fields can be strings or string arrays. When _source is directly set to a string or string array, it’s considered a shortcut for a SourceFilter where only includes is set. With this mechanism, _source can be built in many ways, as Elasticsearch was designed to make it easy to write simple requests by hand, while still allowing complex ones. Shortcut properties are common, so the Elasticsearch specification models them with an annotation, requiring a single line of code. The OpenSearch specification, however, models them again using “oneOf” with two types: one string and the other an object. This has three implications:

  1. Adding a shortcut property requires more than one line change
  2. When reading the specification, it’s not apparent that a shortcut property was modeled
  3. Once again, the Java SDK needs to use a complex set of rules to recognize this case

I could mention other cases of the Elasticsearch modeling guide, but it is possible to generate complex SDKs from an OpenAPI specification. However, this results in a specification that can be more difficult to maintain, as even the most minor changes can break the SDKs, given the API is described implicitly. While custom linters are essential to keep this maintainable, it’s not unlike writing code in assembler instead of C, and then trying to disassemble the result to understand the original intent. To achieve this, code generators must rely on the exact way the specification was written, without any abstraction to help. This is why the Java OpenSearch client could not rely on OpenAPI Generator, and had to write its own OpenAPI implementation. It’s an engineering feat, but is it less effort than using a custom specification?

In this case, it can initially be harder to contribute to the Elasticsearch specification: it's TypeScript, which helps, but it's still custom tooling. Still, it is easier to maintain as many aspects are described explicitly. This is especially true of SDKs for statically typed languages, such as Java, .NET, Rust, or Go, which add many constraints of their own that the specification and generator need to cater to. To make it easier to use for drive-by contributors, we’re currently exploring ESlint custom rules, which give feedback directly in IDEs, without needing any special configuration.

I hope you found this overview interesting! As context, between February and July 2025, I worked on a guest post for the Pragmatic Engineer newsletter explaining how to build SDKs for tech companies, and comparing different approaches to specify those APIs. It was fascinating to see the behind-the-scenes work needed to write high-quality articles, and I don’t know how Gergely produces so much content every week. Anyway, in that guest post, the section that compares the Elasticsearch and OpenSearch API specifications, did not make the final cut. I'm now posting it on my blog with Gergely's blessing. Note, this blog post does assume familiarity with API specifications. I explain these concepts in the Pragmatic Engineer guest post, so don't repeat them here.

I'm on Mastodon!

Comments