The Apache Software Foundation has version 2.1.0 by Apache Atlas. The Open Source Framework is to support HADOOP users in companies in order to fulfill compliance requirements by suitable metadata management. For this, Atlas is among others a number of scaling and expandable governance services. Beyond the error cleanup, the development team has focused on the new release for improvements in performance as well as in the search.
Classify better, search faster
New in Atlas 2.1 are, for example, labels for the so-called entities (metadata types). Users have the possibility to provide entities with matching descriptions – and to remove them again if necessary. When looking for entities not only the new labels are available, but Atlas also offers the option to find them based on more than one classification. To classify, on the one hand, the Framework offers the new feature business metadata, which provides you to describe entitae types over additional attributes closer and filter in search. In addition, Atlas users with Custom Attributes still provide further individual description options for entity instances that have not yet been defined in Entity-DEF or Business Metadata.
Thanks to the Quick Search feature, the search in the framework is already supplying the first suggestion when entering search terms. Meanwhile, more performance promises improvements in the transfer of classifications as well as the tracking of the origin of data (Lineage Retrieval).
The hooks controlled connection of external data sources and systems includes among other innovations for Hive, Spark, AWS S3 and ADLS-Gen2. For Hive, new entities were added: HIVE_PROCESS_EXECUTION makes it possible to track process executions and via HIVE_DB_DL and HIVE_TABLE_DDL can be recorded in DDL operations. New models for Spark, AWS S3 and ADLS-GEN2 enables in Atlas 2.1 to capture entities and relationships from these sources – in the case of Spark also processes can be recorded.
An overview of all improvements and innovations in Apache Atlas 2.1.0 can be found in the release notes. The governance and metadata framework is available for download via the project website of the Apache Software Foundation.