Building a JavaScript SDK for Dremio

Dremio’s REST API

Users can interact with Dremio through a comprehensive set of REST APIs, allowing DevOps teams to orchestrate Dremio with other components of their technology stacks, and allowing end users to more easily build web applications directly on top of Dremio. Some of the operations that you can execute through Dremio’s API are:

• Issue queries
• Retrieve query results as JSON files
• Browsing and managing data catalog
• Manage and refresh reflections
• Check on the status and results for a specific job
• Manage cluster resources
• Manage users and groups
• Manage the Data Reflections’ voting system

You can find the documentation for the API here, and follow along to learn how to perform some basic requests in Python. This example we are going to create a new space.

First, you need to authenticate your user. The Dremio REST API uses a token-based authentication system, so to authenticate a user first we need to send a POST request to our Dremio instance in order to generate the token, the python code for this process looks similar to this:

def login(username, password):
  # we login using the old api for now
  loginData = {'userName': username, 'password': password}
  response = requests.post('http://demo.drem.io:9047/apiv2/login', headers=headers, data=json.dumps(loginData))
  data = json.loads(response.text)
  # retrieve the login token
  token = data['token']
  return {'content-type':'application/json', 'authorization':'_ dremio{authToken}'.format(authToken=token)}
headers = login(username, password)

Then we can use the Post Catalog method to create a Space. The name of our space is “DremioRocks”

import requests
url = "http://localhost:9047/api/v3/catalog"
payload = "{\n    \"entityType\": \"space\",\n    \"name\": \"DremioRocks\"\n}"
headers = {
    'Authorization': "_ dremioo8opojj6vn4ughkvcpalpr46d6",
    'Content-Type': "application/json",
    'cache-control': "no-cache",
    'Postman-Token': "8b1261e4-fab7-439d-ab29-cfce8d3f956f"
    }
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)

Once the code is executed successfully, our new space is generated in the UI.

What are some other interesting things you can do with the API? Here’s another example: the team at DataSprints, a Dremio partner in Latin America, has developed a JavaScript SDK that includes all of Dremio’s API methods and is a great example of how Dremio’s API can make life easier. Now let’s take a look at how this process works with the Dremio JavaScript SDK

Installing and using the Dremio SDK

To install the SDK simply run the following command on your terminal window or command prompt:

$ npm install dremio-sdk

How to use it

First, you have to import the SDK:

$ var Dremio= require('dremio-sdk')

There are three different contexts on which you can add mandatory configurations to the SDK, note that here is where the SDK takes care of the user authentication process:

Through the SDK instance context:

const dremio = new Dremio({
  origin: 'http://test.dremio.local:9047',
  version: '3',
  username: 'dremio',
  password: 'password123'
})

Through the “configure function” context:

const dremio = new Dremio()
dremio.configure({
  origin: 'http://test.dremio.local:9047',
  version: '3',
  username: 'dremio',
  password: 'password123'
})

Or through the service’s context

const dremio = new Dremio()
const Space = dremio.Space({
  origin: 'http://test.dremio.local:9047',
  version: '2',
  username: 'dremio',
  password: 'password123'
})

Usage example - Creating a Space

The following is an example of how to use the SDK to create a new space using two different methods.

Using ‘Promise’

Space.create('DremioRocks').then((spc) => {
  console.log(spc)
}).catch((err) => {
  console.error(err.message)
})

And also using ‘async await’

const createSpace = async () => {
  try {
    return await Space.create('DremioRocks')
  } catch (err) {
    console.error(err.message)
  }
}

Services that are included in the SDK

Conclusion

In this article, we gave a brief introduction to what is an SDK and also what are the roles on the APIs, we also touched on what are the current available Dremio APIs. We shared the benefits of having an SDK that would include all Dremio APIs and also covered details about the SDK that the DataSprints team developed using Dremio’s API information.

Dremio is the data lake engine that allows you to run fast queries on your data lake store, its openness and flexibility means that there is a plethora of compelling ways that you can use it to accelerate your time to insight and help members of your team to be more productive and self-sufficient.