Is this you? As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.
Claim your profile
Articles by Ryan Blue
Unlocking Data Portability: Preventing Catalog Lock-in with REGISTER and UNREGISTER APIs
As organizations embrace the lakehouse architecture, data is shifting from proprietary data warehouses into open storage and table formats, where it can be accessed across multiple engines, like Spark and Trino, without duplication. However, open formats are only one part of the openness equation. True openness also requires interoperability and flexibility in how data is managed and governed.
Read Restrictions and Catalog Labels: Unifying governance across engines and catalogs
In our previous posts, we showed how open table formats, open APIs and unified governance are coming together to complete the Open Lakehouse vision. We also introduced cross-engine attribute-based access control, which allows policies defined in Unity Catalog to be enforced consistently when external engines access governed data. Now, that vision is beginning to materialize in the open.
Unity Catalog and the next era of Apache Iceberg™
The next phase of the open lakehouse will be defined by the catalog. Open table formats made it possible for many engines to work on the same data, but the catalog determines whether that data can be governed, optimized, and shared consistently across systems. As more workloads, including AI and agentic applications, depend on governed access to data across many systems, enterprises need an Iceberg catalog that can provide interoperability, great performance, and enterprise-ready governance.
The Next Era of the Open Lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Today, Databricks’s support of Iceberg v3 enters Public Preview, unlocking the latest innovations from the Iceberg community natively on the open lakehouse. Iceberg v3 marks a major step forward for open table formats, unlocking use cases across incremental data processing and semi-structured data analysis which previously required brittle workarounds.
Advancing the Lakehouse with Apache Iceberg v3 on Databricks
Summary • Databricks supports Apache Iceberg v3, so customers can run interoperable, governed workloads on a single copy of data • With Iceberg v3, deletion vectors, row-level lineage, and the Variant data type are now available on all managed tables • With these features, Databricks brings the Data Intelligence Platform to all formats for the best performance Databricks supports Apache Iceberg v3 in the Data Intelligence Platform, giving customers a unified and open data layer with...
Introducing Variant: A New Open Standard for Semi-Structured Data in Apache Parquet™, Delta Lake, and Apache Iceberg™
Semi-structured data is everywhere in AI, application logs, and telemetry. This data is useful, but changing schemas makes it challenging to store and query. For years, the standard practice was to store this data as strings. Strings were flexible but had poor query performance, since the engine needed to parse and search through the entire string. The Variant data type, now ratified in Apache Parquet™, takes a different approach.
Iceberg v3: Moving the Ecosystem Towards Unification
Iceberg v3 contains major new features (deletion vectors, row lineage, semi-structured data, geospatial types) and unifies the data layer across formats Published: June 2, 2025 Open Source5 min read by Ryan Blue, Daniel Weeks and Aniruth Narayanan Summary Iceberg v3 contains new features and improvements: Deletion Vectors, Row Level Lineage, Semi-Structured Data, and Geospatial Types With these features, Iceberg v3 unifies the data layer across Apache Iceberg™, Delta Lake, Apache Parquet, and...
Apache Iceberg™ v3: Moving the Ecosystem Towards Unification
Apache Iceberg™ v3 contains major new features (deletion vectors, row lineage, semi-structured data, geospatial types) and unifies the data layer across formats Published: June 2, 2025 Open Source5 min read by Ryan Blue, Daniel Weeks and Aniruth Narayanan Summary Apache Iceberg™ v3 contains new features and improvements: Deletion Vectors, Row Level Lineage, Semi-Structured Data, and Geospatial Types With these features, Iceberg v3 unifies the data layer across Apache Iceberg™, Delta Lake,...
Getting ready for the sixth data platform
In the next three to five years, we believe a set of intelligent data apps will emerge requiring a new type of modern data platform to support them. We refer to this as a sixth data platform. In our previous research we’ve used the metaphor “Uber for everyone” to describe this vision, meaning software-based systems that enable a digital representation of a business.
Breaking Analysis: Getting ready for the sixth data platform
With George Gilbert In the next 3-5 years, we believe a set of intelligent data apps will emerge requiring a new type of modern data platform to support them. We refer to this as a sixth data platform. In our previous research we’ve used the metaphor “Uber for everyone” to describe this vision; meaning software-based systems that enable a digital representation of a business.
Building a modern open data lake just got easier with Starburst Galaxy & Tabular
Today, we are excited to announce that the Tabular connector is generally available in Starburst Galaxy. Now, you can get all the benefits of using Trino and Iceberg to build your modern open data lake without worrying about the operational overhead. Background For the past couple of years, Starburst has been sharing Trino’s perspective on the benefits of Apache Iceberg and has seen the adoption of Iceberg in the Trino community skyrocket.
Tutorial: Using Trino and Apache Iceberg for data warehousing
This tutorial demonstrates how to use Trino’s support for Apache Iceberg for data warehousing, including how to useMERGE,UPDATEandDELETEcommands to build and maintain reliable datasets. Prerequisites For this tutorial, you’ll need Trino connected to an Iceberg catalog. If you don’t already have Trino running, the fastest way to try it is usingTabularandStarburst Galaxy.
Tutorial: Connecting Starburst Galaxy to Tabular
This tutorial will help you connect to Tabular from Starburst Galaxy, a managed SQL engine from the creators of Trino. Prerequisites For this tutorial, you’ll need to have aTabular accountand aStarburst Galaxy account. Both services offer free accounts that you can sign up for by clicking the links below: Sign up for Tabular Go toStarburst Galaxyand clickStart Free Tabular In Tabular, you’ll need to create a credential that allows Galaxy to connect to a warehouse.
[VOTE] Graduate to a top-level project
> I would assume because it's a dependancy and not bundled in the source release. I searched for "copied from" to test that assumption and found code from Apache Commons: https://github.com/apache/tomcat/blob/bb0befeebe18a9b8ff50e686a3d06e84ccdfb108/java/org/apache/tomcat/util/http/fileupload/package-info.java#L20-L22 That lines up with my reading of LEGAL-234. If a copyright line for each project was required in NOTICE, then the question of which dates are required would not be relevant.
[VOTE] Graduate to a top-level project
> You see in that Mark recommends including the project name/copyright line in the NOTICE file as I’ve suggested to you. Are you sure about that? Mark is talking about what they did for the Tomcat NOTICE, with no mention of Commons: https://github.com/apache/tomcat/blob/master/NOTICE And the result was this Accumulo NOTICE: https://github.com/apache/accumulo/blob/master/NOTICE On Wed, May 20, 2020 at 3:56 PM Justin Mclean <justin@classsoftware.com> wrote: > Hi, > > > 1.
[VOTE] Graduate to a top-level project
To summarize, it looks like there are two issues: 1. Should the entire file header be moved into NOTICE when a third-party file is contributed, or should we edit it 2. When including all or part of another ASF project is the standard NOTICE sufficient? I went back through legal-discuss and found that #2 has already been asked and answered on https://issues.apache.org/jira/browse/LEGAL-234. The recommendation from Mark Thomas was to not copy the standard NOTICE contents from other projects.
[VOTE] Graduate to a top-level project
On Mon, May 18, 2020 at 4:40 PM Justin Mclean <justin@classsoftware.com> wrote: > > Per NOTICE policy > > <https://www.apache.org/dev/licensing-howto.html#mod-notice>: "Copyright > > notifications which have been relocated, rather than removed, from source > > files must be preserved in NOTICE” > > Yes the copyright line needs to be mentioned not the licenser header. > At the time, we considered the whole header to be the relevant "Copyright notification". Do you think this practice is incorrect?
[VOTE] Graduate to a top-level project
On Mon, May 18, 2020 at 3:35 PM Justin Mclean <justin@classsoftware.com> wrote: > Hi, > > > What license information are you referring to? > > " Licensed under the Apache License” by Cloudera > This is not a license, it is the original copyright header for files that were contributed by Cloudera. Here's the commit <https://github.com/apache/parquet-mr/commit/ddbeb4dd17d9c219b99b1e66d8be28efe37e3aa6> .
[VOTE] Graduate to a top-level project
Hi Justin, I have more questions for you inline: On Sat, May 16, 2020 at 6:29 PM Justin Mclean <justin@classsoftware.com> wrote: > Hi, > 1. You have included some license information this should be in LICENSE > not NOTICE > What license information are you referring to? > 2. Going on what is listed in the LICENSE file you are missing content > from bundled code NOTICE files [1] (see bundling an apache 2.0 licences > dependancy) > Which entries in the LICENSE file?
[VOTE] Recommend Apache Iceberg graduation to top-level project resolution to the board
Hi everyone, With the support of our mentors (as well as helpful ASF members), the Apache Iceberg community has voted to graduate to a top-level project. I propose a vote to recommend graduation for the Iceberg community to the board.
[VOTE] Recommend Apache Iceberg graduation to top-level project resolution to the board
+1 (binding) On Fri, May 15, 2020 at 5:39 PM Ryan Blue <blue@apache.org> wrote: > Hi everyone, > > With the support of our mentors (as well as helpful ASF members), the > Apache Iceberg community has voted to graduate to a top-level project. > > I propose a vote to recommend graduation for the Iceberg community to the > board.
[VOTE] Graduate to a top-level project
Thanks for taking a look, Justin. I appreciate playing devil's advocate here to ensure this is well thought through. I'll add replies inline. On Tue, May 12, 2020 at 10:44 PM Justin Mclean <justin@classsoftware.com> wrote: > Has the project filled in a maturity model? This is optional but give the > project and the IPMC more information about the project.
[VOTE] Graduate to a top-level project
Forwarding the graduation vote thread from the Iceberg community to let everyone here know, per [1].
[RESULT] [VOTE] Release Apache Iceberg 0.8.0-incubating RC2
With 4 +1 votes and no -1 or -0 votes, this passes. Thanks for taking a look at this release, everyone! On Wed, May 6, 2020 at 11:26 AM Owen O'Malley <owen.omalley@gmail.com> wrote: > +1 (binding) > > On Mon, May 4, 2020 at 8:56 AM Ryan Blue <rblue@netflix.com.invalid> > wrote: > >> Thanks, Justin. Can you be more specific about what you think isn't >> correct >> in the NOTICE file? Our practice is to copy the entire notice of >> third-party projects, if they exist.
[VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Thanks, Justin. Can you be more specific about what you think isn't correct in the NOTICE file? Our practice is to copy the entire notice of third-party projects, if they exist. We don't do this for the boiler-plate ASF NOTICE, but any ASF notice that contains other text will be copied in (like Yetus in the runtime Jar).
[VOTE] Release Apache Iceberg 0.8.0-incubating RC2
Hi everyone, I propose the following candidate to be released as the official Apache Iceberg 0.8.0-incubating release. The Iceberg PPMC vote passed <https://lists.apache.org/thread.html/r28e30803aeab81bf822750aa26cb35b378bacd9aa0d68ea463dbcfff%40%3Cdev.iceberg.apache.org%3E> with 12 +1 votes, 7 of which were binding PPMC votes, and 3 of which were binding IPMC votes from project mentors (blue, omalley, and cws).
[VOTE] Release Apache Iceberg 0.8.0-incubating RC2
For anyone that would like to vote on this release that is unfamiliar with Iceberg, here are the community’s recommended checks: 1. Download the source tarball, signature (.asc), and checksum (.sha512) from https://dist.apache.org/repos/dist/dev/incubator/iceberg/apache-iceberg-0.8.0-incubating-rc1/ 2. Import gpg keys: download KEYS <https://dist.apache.org/repos/dist/dev/incubator/iceberg/KEYS> and run gpg --import /path/to/downloaded/KEYS (optional if this hasn’t changed) 3.
[RESULT] [VOTE] Release Apache Iceberg 0.7.0-incubating RC4
This release vote passes, with 5 +1 votes and no +0 or -1 votes. +1 votes: James Taylor Ryan Blue Jacques Nadeau Julien Le Dem Carl Steinbach Thank you for taking the time to verify this release! On Wed, Oct 23, 2019 at 2:36 PM Carl Steinbach <cws@apache.org> wrote: > +1 (binding) > > On Tue, Oct 22, 2019, 4:42 PM Julien Le Dem <julien.ledem@wework.com > .invalid> > wrote: > > > +1 (binding) > > > > Downloaded, verified signature, checked licences, ran the build.
[VOTE] Release Apache Iceberg 0.7.0-incubating RC4
Hi everyone, I propose the following RC to be released as official Apache Iceberg 0.7.0-incubating release. This candidate has passed a PPMC vote.
[VOTE] Recommend 'Apache Druid graduation to Top Level Project' resolution to the board
+1 Congratulations! On Tue, Aug 13, 2019 at 2:03 PM Myrle Krantz <myrle@apache.org> wrote: > +1 binding I'm looking forward to welcoming you to the fold! : o) > > Best, > Myrle > > On Tue, Aug 13, 2019 at 9:10 PM Suneel Marthi <smarthi@apache.org> wrote: > > > +1 binding > > > > Congrats !! > > > > On Tue, Aug 13, 2019 at 3:06 PM Julian Feinauer < > > j.feinauer@pragmaticminds.de> wrote: > > > > > +1 (binding) > > > > > > Congratulations! > > > > > > Julian > > > > > > > > > --------...
[IP CLEARANCE] Apache Arrow Parquet Rust Library
+1 (binding) On Thu, Dec 13, 2018 at 8:24 AM Wes McKinney <wesmckinn@gmail.com> wrote: > Apache Arrow is receiving a donation of a Rust library to read and > write the Apache Parquet file format, intended to work together with > the Arrow Rust library [1]. > > Please vote to approve this contribution. > > This is a lazy consensus majority vote, per the IP clearance process > [2], open for at least 72 hours.
[RESULT] [VOTE] Accept the Iceberg project for incubation
The vote passes with 13 binding +1 and 5 non-binding +1 votes. Thank you for voting, everyone! I'll get started with the next steps. +1 votes: Ryan Blue* Matt Sicker* Felix Cheung Dave Fisher* Owen O'Malley* Hugo Louro Arthur Wiedmer Julian Hyde* Kevin A.
[VOTE] Accept the Iceberg project for incubation
Quick update: James Taylor has offered to mentor the project as well, so I've added him to the list. Thanks, James! On Tue, Nov 13, 2018 at 9:06 AM Ryan Blue <blue@apache.org> wrote: > The discuss thread seems to have reached consensus, so I propose accepting > the Iceberg project for incubation.
[VOTE] Accept the Iceberg project for incubation
+1 (binding) On Tue, Nov 13, 2018 at 9:06 AM Ryan Blue <blue@apache.org> wrote: > The discuss thread seems to have reached consensus, so I propose accepting > the Iceberg project for incubation. > > The proposal is copied below and in the wiki: > https://wiki.apache.org/incubator/IcebergProposal > > Please vote on whether to accept Iceberg in the next 72 hours: > > [ ] +1, accept Iceberg for incubation > [ ] -1, reject the Iceberg proposal because . . .
[DISCUSS] Iceberg proposal for incubation
It sounds like there is consensus, so I'll start a vote thread. Thanks, everyone, for looking at the proposal! rb On Tue, Nov 13, 2018 at 1:27 AM Lars Francke <lars.francke@gmail.com> wrote: > I've been following the Iceberg project on and off for a while now and I > think this is desperately needed. I don't claim to understand all of the > details but I'm very happy to hear this being proposed for Apache. I think > it'd be a great fit. Thank you Ryan and everyone else working on this so > far.
[VOTE] Accept the Iceberg project for incubation
The discuss thread seems to have reached consensus, so I propose accepting the Iceberg project for incubation. The proposal is copied below and in the wiki: https://wiki.apache.org/incubator/IcebergProposal Please vote on whether to accept Iceberg in the next 72 hours: [ ] +1, accept Iceberg for incubation [ ] -1, reject the Iceberg proposal because . . .
[DISCUSS] Iceberg proposal for incubation
Since it has been a week without substantial comments or changes to this proposal, I've posted it to the wiki as Bertrand suggested: https://wiki.apache.org/incubator/IcebergProposal Any further comments or discussion? If not, I can start a vote thread tomorrow. Thanks, everyone! rb On Tue, Nov 6, 2018 at 12:10 AM Bertrand Delacretaz wrote: > Hi, > > On Mon, Nov 5, 2018 at 7:04 PM Ryan Blue > wrote: > >...
[DISCUSS] Iceberg proposal for incubation
Thanks for pointing that out, Dave. That section was accidentally not updated after I copied the markup template. I'll fix it. On Mon, Nov 12, 2018 at 12:01 PM Dave Fisher wrote: > Hi - > > Looks very interesting. > > I have questions about the discrepancy between the Initial Committer list > of 5 people vs. the Affliations list with 16 people? > > Is it true that the podling plans to start with just 5 committers? > > Who from this group is affiliated with Netflix?
[VOTE] Apache Toree 0.3.0-incubating (RC1)
+1 (binding, carried over from the podling vote) On Thu, Nov 8, 2018 at 7:54 PM Justin Mclean wrote: > Hi, > > +1 (binding) > > I checked: > - incubating in name (although I notice the pip package doesn’t - is this > a limitation?) > - signatures and hashes all good > - DISCLAIMER exists > - LICENSE and NOTICE file > - all source files have ASF headers > - No binary files in release > - Can compile from source > > Thanks, > Justin >...
[DISCUSS] Iceberg proposal for incubation
Hi everyone, I'd like to start a discussion about moving Netflix's Iceberg project to the incubator. Iceberg is a library and specification for tracking data files that store table data in the big data ecosystem. Iceberg is designed to guarantee snapshot isolation and improve on performance problems with big data tables, especially when using S3 or another object store as the source of truth. We've written up an Iceberg proposal in google docs.
[IP CLEARANCE] Arrow Gandiva library
+1 On Mon, Sep 24, 2018 at 9:24 AM Matt Sicker wrote: > +1 > > On Fri, 21 Sep 2018 at 14:09, Wes McKinney wrote: > > > Apache Arrow is receiving a donation of the Gandiva library, an > > LLVM-based analytical expression compiler framework [1] > > > > Please vote to approve this contribution. > > > > This is a lazy consensus majority vote, per the IP clearance process > > [2], open for at least 72 hours.
Late Toree Podling Report
I added my sign-off on the report. Thanks for putting it together, Luciano! On Sun, Aug 12, 2018 at 4:36 PM Justin Mclean <justinmclean@me.com> wrote: > Hi, > > Added and I assumed and marked it signed off by you. > > Thanks, > Justin > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org > For additional commands, e-mail: general-help@incubator.apache.org > > -- Ryan Blue Software Engineer Netflix
Looking for Champion
Okay, then let me rephrase: I would like to see a plan in the Palo proposal for a licensing scrub to be done before graduation. I'm still a little skeptical about this practice because the Incubator PMC validates the release on behalf of the foundation, but I think that's a separate issue to consider that doesn't need to distract on this Palo thread.
Looking for Champion
> we have allowed (and IMO should continue) podlings to have licensing issues during their incubator releases Thanks for pointing this out, Greg. I wasn't aware of this and have always had releases fail when we discover licensing issues. I think there's a significant risk of license problems, so I had assumed we would require a thorough scrub before the first release. What's the argument for finishing this work before graduation rather than first release?
Looking for Champion
I agree with Jim, at least mostly. I don't mind code and toil duplication between projects in itself, but I think that the current state of the project shows that there are two large risks to the potential Palo podling (for lack of a better name): 1. The choice not to work with the Impala community initially shows a risk of not working with others when it may be more difficult to do so than not.
Introducing Iceberg: Tables Designed for Object Stores - Strata Data Conference in New York 2018
Who is this presentation for? Big Data Software Engineer or Data Scientist The attendee should have a familiarity with big data processing in Hive, Spark, or Presto. In this talk, we'll cover the motivation, design, and show some performance metrics of the new tables, which can be 10x-100x faster for query planning. Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic.
[VOTE] Apache Toree 0.2.0-incubating (RC4)
How did you run tests? It worked fine from source for me with `make test` On Fri, Apr 13, 2018 at 1:12 PM, Matt Sicker wrote: > I built it from the source tar.gz.
[VOTE] Apache Toree 0.2.0-incubating (RC4)
For the .md5 file, we can simply not publish that file when copying the RC files. Please don't let that prevent you from checking this release! On Wed, Apr 11, 2018 at 8:41 AM, Luciano Resende wrote: > Here is my binding +1, any volunteers to help review the RC, we need one > more IPMC vote. > > On Sat, Apr 7, 2018 at 6:28 PM, Luciano Resende > wrote: > > > Please vote to approve the release of Apache Toree 0.2.0-incubating > > (RC4).
[VOTE] Accept Druid into the Apache Incubator
+1 (binding) On Thu, Feb 22, 2018 at 11:24 AM, Ashish wrote: > +1 (non-binding) > > On Thu, Feb 22, 2018 at 11:03 AM, Julian Hyde wrote: > > Hi all, > > > > After some discussion on the Druid proposal[1], I'd like to > > start a vote on accepting Druid into the Apache Incubator, > > per the ASF policy[2] and voting rules[3]. > > > > A vote for accepting a new Apache Incubator podling is a > > majority vote for which only Incubator PMC member votes are > > binding.
[VOTE] Apache Toree 0.2.0-incubating (RC3)
Justin, A bit more context on those Jars: these were created to test the ability to add jars at runtime to the Scala interpreter, and they were contributed to the project as tests. I'm not sure there's a strong case to build these Jars just before using them in tests. That would require much more time and maintenance than just including the binaries. There's nothing special about them besides that they have classes that can be loaded to verify the classpath addition worked.
Show More
loading
Actions
Is this you?
As a journalist, you can create a free Muck Rack account to customize your profile, list your contact preferences, and upload a portfolio of your best work.Get in touch with Ryan
Contact Ryan, search articles and posts on X, monitor coverage, and track replies from one place.
Learn more about Muck Rack