I sat down with Bianca Lewis of OpenSearch at Open Source Summit India, right after the project's second OpenSearch Con in the country. Bianca Lewis is the executive director of the OpenSearch Software Foundation.
OpenSearch started life in 2021 as a fork of Elasticsearch, born out of Elastic's move away from an open source license. Five years on, I wanted to know how much of that origin story still holds, and how the project sees itself now that it's an AI data infrastructure layer with its own roadmap and its own community.
We covered a lot of ground: whether OpenSearch is still tethered to Elasticsearch in any way, why ELK Stack remains the more recognizable brand among young developers, how agentic AI fits into the platform, and an uncomfortable but necessary question about what happens when vendor-backed open source projects go proprietary.
Five years in, is OpenSearch still a fork of Elasticsearch, or has it become a product of its own? Does it still pull in any downstream changes from Elastic?
Bianca: That is a great question, and the answer, very simply, is yes, it's an independent project. It's a completely independent project now that began as a fork, as you correctly said, and in the early days the two were very, very similar. Today the focuses and the strengths have become very different, and today they are completely independent projects. I think the only commonality is that users can still use both platforms for similar kinds of use cases.
Explaining OpenSearch to the average Linux user
OpenSearch is enterprise-focused, but how would you explain it to a home labber or self-hoster who's never heard of it?
Bianca: Well, firstly, of course OpenSearch is an AI infrastructure layer for enterprises, but because it's on an open source platform, free and easy to use and adopt, people making new startups, entrepreneurs, developers, and everyone else find it very easy to get started. Download it, it's free to use, all the features are open and available, and they can join very easily a community of thousands and thousands of people to get up and running.
So it's not just enterprise-focused anymore, it's for anyone and everyone. Whoever's requirements fit it, OpenSearch is for them. I don't think it was ever built to just be for the enterprise. The startups of today are the enterprises of tomorrow.
"The startups of today are the enterprises of tomorrow."
The ELK Stack brand problem
ELK Stack is still the more recognizable brand in observability and search. How does OpenSearch compete with that brand recognition among young developers?
Bianca: There is no doubt that Elastic, Elasticsearch, and Logstash and Kibana, which make up ELK, have been around a long time, and that's obviously where the fork came from originally. I hope that after five years of OpenSearch developing as an independent and powerful project — 1.7 billion downloads later, 400 individual companies contributing, more than 3,000 contributors, and 15,000 OpenSearch stack users — the differentiation in young developers' minds is becoming clearer just from the sheer weight of numbers.
I think there are two distinguishing factors. The first is philosophy — how developers want to build their technology stacks conceptually and into the future. Elastic, who owns ELK, even though the license is now back to open source after they closed it, is still owned by Elastic and is open core. That means if a young developer wants to build on Elasticsearch and use the ELK stack, they can use the core version for free, but for any more advanced features or support they're going to have to pay and be tied into Elastic. Developers who want to build on a platform that's fully open, where you get the whole thing for free and can choose whether you want support or hosting when you're ready, have that choice with OpenSearch. It's a different philosophy.
The second layer is that the two platforms have grown along different technological paths. Every platform in this space will say "we're a millisecond faster at this," or "we support this use case better," and that's all true — no platform can honestly say it's the best at everything. But OpenSearch is really good at a lot of different use cases that unify the data layer, and because it's the only alternative that isn't just open core, it's making young developers distinguish OpenSearch, hopefully much faster than ELK or any other platform.
Observability, complexity, and cost
Observability stacks are getting heavier, more expensive, and more complicated. What is OpenSearch doing about ease of setup and cost?
Bianca: Those are two very valid questions, so let me answer them one by one.
On complexity: the advantage of proprietary vendors has always been that on day one it's easy — you buy it, it's off the shelf, it works, and you get support. With an open platform, you build it on day one, and you get the benefit on day two, three, and four because it's customized to your needs and you're not dependent on a vendor. But getting started on day one is a very valid point. Because of that, we've recently launched something called Launchpad, which lets you eailylaunch an observability stack, a search application, or any other OpenSearch application, because OpenSearch isn't just observability, there's also search, AI applications, and security monitoring. All this takes a few minutes, and we encourage developers to ask the community if they get lost, even though the documentation is all there. I believe the ease of setting up OpenSearch has been addressed incredibly well and quickly.
On cost, it's a more complicated question. I can tell you OpenSearch is free, but in real life nothing is ever truly free because you still have to run it on infrastructure and manage it.
What OpenSearch does do is make everything cost-effective. Take observability: in the old days, we had to have it because if something in our infrastructure broke, we had to solve it — we couldn't afford downtime — but nobody wanted to pay for it because it was seen as a cost center.
Today the whole language has changed, because OpenSearch gives you a unified view of the context of your data, which is much more cost-effective than doing log analytics, traces, and metrics separately with tools like Grafana and Prometheus, where everyone looked at the data differently and root-cause analysis took five weeks to put the pieces together.
Today it's all in a single pipeline into OpenSearch — you see it all together, through different lenses in the same view, and you can automate resolutions. You can use models and AI to interrogate that data, automate resolutions, and use your observability stack for more than troubleshooting — for company-wide SLOs, understanding acceptable error rates, and understanding cost.
We're in the token economy now, so we can add cost information into traces and logs and understand accurately how much each service is really costing us. Observability isn't just "pay for troubleshooting" anymore — it's become easier to use and a driver of business imperatives.
AI-agent readiness
How AI-agent-ready is OpenSearch, for someone building a project that leans heavily on agentic AI?
Bianca: In the 3.5 release — we've just released 3.7, so we're two minor versions past that — agentic AI became generally available in OpenSearch, and since then it's become a central theme of how people run their search and observability stacks. I think we're ahead of almost any other platform in the world on fully integrated agentic AI.
As you spin up AI agents, OpenSearch is model-agnostic, so you can use any AI model you want. If it's for a search use case, you can monitor those AI agents with the same platform — monitor the trace of the agent, see which services it's calling, what the costs are, and make sure it's doing the right thing. At scale, when you're running 100,000 queries a minute, we can almost let AI monitor AI for security, safety, and compliance.
AI brings scale and the ability to do things quickly. We don't have to work sequentially anymore. But data without context is just data, it's noise. You need a platform that can recall it and spin up AI agents, you need the data underneath it, and you need the agents to talk to each other to perform tasks but without context, all of that is garbage at scale. OpenSearch has built in some features to address this: short-term agentic memory, so an agent remembers the context of what it was asked over the last week or two, and in 3.7, long-term agentic memory as well, which gives us the context we need to take accurate actions on the data. It's already generally available.
AI data layer vs. traditional search
Is OpenSearch now an AI data layer sitting on top of a traditional search stack, or have the two fully merged?
Bianca: I think ultimately OpenSearch will be looked at in a couple of layers. OpenSearch is the application level, and the application level sits on top of an infrastructure that supports the architecture of the application.
What OpenSearch gives you is that it doesn't tie you into any one vendor's choices. You can have that AI infrastructure layer but choose your own infrastructure. You can self-host, or go to one of the hosted services, like our OpenSearch service on NetApp, Oracle, and AWS and choose whatever fits your model. In terms of features and how things run with agentic AI, we don't really say this works better or worse than that. We say everything works and works really well, and with OpenSearch you retain control over how you want to build it, without being tied into any one approach.
On business-source licensing and the Linux Foundation model
This is a bit of an uncomfortable question, more about the Linux Foundation model in general than OpenSearch specifically. MariaDB, Redis, and Elasticsearch all moved from open source to business-source or server-side licenses at some point, largely to protect their business from hyperscalers who weren't contributing enough but were taking their enterprise customers. Then the Linux Foundation and companies like Amazon fork the original project, as happened with Terraform and OpenTofu, and grow it as the "true" open source version while the original gets left behind. Is there an injustice in that, toward the original projects?
Bianca: I think it's a really great question. I don't ever judge a vendor for deciding to close-source or open-source a project. But without intruding into the particulars of how Elastic or AWS felt at the time, which I can't speak to, I can share a personal viewpoint.
If a vendor is the owner of an open source project, to me that open-core label is simply a go-to-market strategy to get users in and then sell services, support, and enterprise capability. It's not what we'd call vendor-neutral. Building a wide community on a vendor-owned open source project is always tricky, and that's why Mongo, Elastic, Redis, and a few others reached a point where the situation became a threat — the go-to-market had been really successful, they'd built a community, but the biggest customers weren't converting.
I find it interesting that Elastic changed back to an open source license after closing it — because they were losing the community, but they'd closed it because they were losing enterprise customers. They were fine with small players using the free product but wanted the big-money customers. The challenge is that often you want your cake and to eat it too. Once you build your brand on community involvement and then tell the community you're closing it, even if you reopen it later, a lot of the community will just say "thanks, but no thanks."
That's the beauty of the Linux Foundation model. It's a nonprofit that nobody owns, and profit isn't the motive. Yes, there will be cases where hyperscalers fork a project. I think it's important to understand that once they fork it and give it to a nonprofit like the Linux Foundation, they give up ownership. They don't own OpenSearch anymore. They could fork it again and keep something in-house, anyone can always do that, but OpenSearch itself can never be closed now, because the Linux Foundation owns it.
OpenSearch itself can never be closed now, because the people that own it is the Linux Foundation.
It also helps to look at where each company actually makes its money. Take Elastic. They make money selling Elasticsearch licenses, whether self-hosted or on the cloud, so it makes sense for them to gate advanced features behind a license, because they have to survive as a business and answer to shareholders.
Hyperscalers like AWS, Microsoft, Google, IBM, and Oracle make their money from selling compute and infrastructure, not directly from the application. Carrying the burden of growing and innovating an application takes away from their core focus of selling infrastructure, so it makes business sense for them to let the community develop the application while they focus on providing a great service to run it, and put their money where they actually make money.
That's the subtle reason hyperscalers are so eager to support Linux Foundation projects. It lets them focus on where they make money, and it guarantees these projects stay open, available, and well-maintained rather than becoming orphaned.
Ultimately it takes time for a project's original founder to say "now we can let the community manage it." What that actually means is that over the years, instead of maintaining the application with a full-time in-house engineering team, the founder becomes one of the companies contributing to a shared engineering effort. You get all the benefit, but the innovation and cost become shared, and the community retains the choices, which is always good for business.
Certifications
Does OpenSearch have plans for a certification track, like Kubernetes has with the CKA?
Bianca: Yes, we've actually recently announced that. The way we're doing it is quite unique and exciting for the open source world. We've already released the first form of learning content, and in a couple of months we're going to release an OpenSearch competency course that carries a badge of competency you can put on LinkedIn.
On top of that, we're going to do the OpenSearch competency certification proper, and based on that certification, we're doing accreditations for different vendors and companies. The follow-up to that is specialist certifications. Since OpenSearch is used across quite a few use cases, we'll have competencies in observability, search, AI applications, and security monitoring. And to renew your certification, instead of retaking the same test every couple of years, as long as you're advancing to more advanced certifications you keep your original qualifications. We want people to grow and advance within OpenSearch.
The biggest misconception about OpenSearch
What's the one thing that most people get wrong about OpenSearch?
Bianca: There are so many things people get wrong about OpenSearch, but I think the biggest one is not recognizing that, as an AI data infrastructure layer, OpenSearch is the only cloud-native, AI-native platform in the world that's truly open source and vendor-neutral. I don't know if people really understand that.
OpenSearch is the only cloud-native, AI-native platform in the world that's truly open source and vendor-neutral.
Maybe if it were named something like "OpenSearch AI" or "OpenAI Search," it would be more recognizable.
Bianca: Yeah because people often equate it to "how does it compare to Elasticsearch?" But conceptually, you can't really compare the two. They'll say "it's like the AWS service," but that's just a vendor who's a member and provides a service on top of it. The platform runs completely independently, within a community that's truly open source. It's not just an AWS project anymore; it's a truly independent, community-owned observability stack.
OpenSearch's 3.7 release, with long-term agentic memory, is already out, and the project's competency certification track is expected to roll out over the coming months. You can find the project's documentation, downloads, and community channels at opensearch.org.
from It's FOSS https://ift.tt/gsK6Ah2
via IFTTT
Tidak ada komentar:
Posting Komentar