Announcing Dremio Hub

Announcing Dremio Hub

Today we are announcing Dremio Hub – a marketplace of community-supported connectors for a broad variety of data sources, plus the ability to create your own.

Dremio provides lightning-fast queries on Data Lake storage and on a large variety of other data sources. In addition to the native connectors built into Dremio, Dremio Hub provides a marketplace of additional community provided connectors to download, making it easy to join your Data Lake storage with all other places you keep your data, without ETL.

Dremio Hub is launching with connectors for Snowflake, Salesforce, and several other popular data sources. Installing Dremio Hub connectors is easy, and we expect to have many more connectors coming over the next year.

Dremio Hub Marketplace

The Dremio Hub Marketplace provides an accessible and easy to use listing of Dremio Community provided connectors to a wide variety of data sources, from Relational Databases through SaaS applications. Simply find the data source to connect to, download the connector and follow the straightforward installation steps.

After installation Dremio Hub connectors appear as new data sources within Dremio the same as standard connectors. Simply click on the new source and follow the same configuration steps as for standard Dremio connectors.

Contribute

In addition to providing a marketplace of connectors for download, Dremio Hub includes a easy to use framework for Dremio Community members to build new connectors and to share and publish their connectors for the Dremio Community. Connectors can be built to any data source with a JDBC driver and are template based making it simple and easy to define new connectors without complex coding required.

Dremio Hub connectors have the exact same high-performance capabilities as native Dremio connectors, including Advanced Releational Pushdown which executes complex SQL logic directly within the data source for high-performance and the ability to define Reflections.

To contribute a new connector to be hosted on the marketplace follow the instructions to develop and build new connectors, and then submit the requested information and connector components to [email protected], either as an attachment or via cloud storage such as Azure ADLS, Amazon S3 or Dropbox.

Additional API Interfaces

In addition to source connectors to relational sources, examples of internal API interfaces are available for advanced users to extend the functionality of their Dremio instances in a variety of areas. These interfaces include:

• Base API for data sources - The Base API enables users to create high performance parallel connectors to external data systems and can be used to define connectors for a wide variety of systems. An example HBase connector using the API is available on GitHub here.
• Arrow Flight API - Arrow Flight provides an RPC/IPC interchange library for efficient & parallel interchange of data between systems that offers 100x to 1000x higher performance than ODBC/JDBC. An example Arrow Flight driver is available on GitHub here.
• User-defined Internal Function API - The User-defined Internal Function API enables users to develop their own custom functions for using in SQL queries. An example function is provided on GitHub here.

Dremio Hub enables the Dremio community to customize and extend the functionality of Dremio for a wide variety of situations and use cases, we are excited to offer this new capability and look forward to community contributions to the marketplace.

Stay tuned for more news!