Introducing Elastic Engines - Dremio
Introduction
Dremio AWS Edition supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, start and stop based on predefined workload requirements at runtime. This provides several benefits, including:
- Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
- Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
- Track cost by team by running workloads on their own resources
- Right size execution resources for each distinct workload, instead of implementing a one size fits model
- Easily experiment with different execution resource sizes, at any scale
In this article we walk you through the steps to provision and manage Elastic Engines, we also show you the steps to manage workloads using queues and rules.
How to Launch and Manage Elastic Engines?
Step 1. Elastic engines is one of the easiest features to use in Dremio. To provision a new engine, in the Dremio UI go to** Admin -> Elastic Engines**
Step 2. You will see a default engine already deployed. Now click on Add New
Step 3. The Set Up Engine popup page by default automatically selects the EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). To continue the configuration, add a name, select the instance count and then click on Advanced Properties(optional)
Step 4. Review the Advanced Properties menu. Here is where we will have the option to enable Auto Start and Stop.
- When Auto Start is enabled: The engine will start as soon as it receives the first query.
- When Auto Stop is enabled: The engine will stop after 5 minutes of inactivity.
Once you are ready selecting your options, click on Save & Launch
Workload Management
Creating a Queue
Now, we need to tell Dremio which workloads we want it to process using any of the existing engines. To do so follow these steps:Step 1. Navigate to the Queues option within the Workload Management **in the **Admin settings. Then click on Add New.
Step 2. Give the new queue a name, then from the Engine Name drop-down menu select the engine that you just created (or the engine that you want to assign to this queue). Here you will also see options related to time, memory and CPU management. When done, click Save.
Creating a Rule
Now, we need to create a rule where we will specify that if a workload (or query) meets certain criteria, it will be handled by a certain engine. To create a new rule follow these steps
Step 1. From the Admin menu, click on Rules and then Add New.
IMPORTANT: The priority of the rules is defined by their order i.e rule 1 will be parsed first and can supersede any other rules that follow. To change the order of the rules, click on three bar icon next to its name and drag it to the desired position.
Step 2. Give the rule a name and conditions. You will see several condition examples that you can use i.e. User, Group Membership, Job Type, Query Plan Cost, or a combination of all. Once you add the condition, you have the option to reject or assign the workload to the queue that you created in the previous section.
This means that any workload that meets the condition requirements, will be assigned to a queue, and then that queue will be executed by a specific engine.
Step 3. When done, simply save the queue and launch a query to test out the new configuration. Visit our documentation to learn more details about how to properly configure workload management queues and rules within Dremio.
Wrapping Up!
To learn more about Dremio visit our tutorials and resources, also if you would like to experiment with Dremio in your own virtual lab, go ahead and checkout Dremio University, and if you have any questions visit our community forums where we are all eager to help.
