Introduction
The Morpheus Plugin API is a Java 8 based library for creating Plugins that add functionality to Morpheus. The Plugin api supports implementing providers to Morpheus of the following types:
-
UI Extensions
-
Task Types
-
IPAM
-
DNS
-
Approvals
-
Cypher Modules
-
Custom Reports
-
Cloud Providers
-
Network Providers
Release Notes
Note: Morpheus Plugin API is now 1.0! This means calls that are used will be supported for a longer period of time and given appropriate deprecation warnings when necessary. Morpheus Plugin API 1.3.x requires a minimum Morpheus version of 8.1.x. Morpheus Plugin API 1.2.x requires Morpheus 8.0.x, and Morpheus 7.0.x runs on Core version 1.1.x.
-
1.3.4
-
File Copy Enhancements
-
Added
FileCopyRequestsupport and newgenerateUrloverloads for file copy operations with Cloud context.
-
-
Network Pool and Floating IP Enhancements
-
Added network pool lifecycle hooks to
IPAMProviderfor create, update, delete, and validation workflows. -
Added
optionTypessupport toNetworkPoolTypeandNetworkFloatingIpPoolType, plusFloatingIpProvider#getOptionTypes()for custom configuration fields. -
Added floating IP pool CRUD and validation methods to
FloatingIpProvider, and addeddescriptiontoNetworkFloatingIpPool.
-
-
Backup, Console, and Localization Improvements
-
Added synchronous
updateStatussupport toMorpheusSynchronousBackupProviderService. -
Added
executionTypetoBackupResult. -
Added typed hypervisor console support with
ProvisionProvider.HypervisorConsoleFacetV2andHypervisorConsoleConnection, while deprecating the legacy map-based console contract. -
Added
MorpheusLocalizationServiceoverloads with default value support.
-
-
-
1.3.3
-
Datastore Provider Validation
-
Added
validateRemoveDatastoretoDatastoreTypeProviderso plugins can validate datastore removal requests before delete operations (MORPH-6732).
-
-
-
1.3.2
-
Release and Publication Updates
-
Streamlined release workflow configuration.
-
-
-
1.3.1
-
Cloud and Credential Validation
-
Improved
ValidateCloudRequestto support all credential types (MORPH-760).
-
-
Storage and Volume Enhancements
-
Added
filterStorageVolumeTypesmethod to provider interfaces for volume type filtering (MORPH-4070). -
Added
storageProviderfield toVirtualImagemodel class. -
Added
defaultBucketNameanddefaultStorageProvidertoImportWorkloadRequest. -
Added storage provider documentation.
-
-
Backup and vCenter Support
-
Added missing model fields and backup service methods for vCenter plugin support (MORPH-757).
-
-
Affinity Group and Resource Management
-
Updated
AffinityGroupsto support full CRUD operations.
-
-
Network Improvements
-
Minor model changes for floating IP pool support.
-
-
Provider Versioning
-
Updated
@sinceannotations to1.3.1forDatastoreTypeProviderandClusterProvider.
-
-
-
1.3.0
-
Morpheus 8.1.x Compatibility and Generator Updates
-
Added project template generation support for Morpheus
8.1.x. -
Added
8.1.xto the plugin generator Morpheus version dropdown. -
Updated generated Gradle templates to use
morpheus-plugin-api:1.3.0andmorpheus-plugin-gradle:1.3.0.
-
-
Networking and Security Group Enhancements
-
Added router tab provider support for plugin UI rendering.
-
Added security group rule target support and default target types for easier provider implementation.
-
Added support for NetworkLocation manipulation and related network provider capabilities.
-
-
Storage and Compute Model Improvements
-
Added
isMultiAttachsupport toStorageVolumeand aligned default handling. -
Added support for
StorageAggregates. -
Added support for affinity groups.
-
Added additional ComputeServer model and provider enhancements, including server stats and layout override support.
-
-
Platform and Service Updates
-
Added missing
MorpheusSynchronousDataQueryService. -
Updated selected model date types from
java.time.Instanttojava.util.Datefor core compatibility. -
Added support for costing/reservations and account price history related model updates.
-
-
-
1.2.13
-
Storage and Datastore Enhancements
-
Added checkpoint target capability to datastore type provider for VM snapshot management
-
Added support for asynchronous datastore type and location operations
-
Implemented prepareVolumeAttach for datastore type providers
-
Added StorageHost and StorageHostGroup services with comprehensive models
-
Enhanced StorageHost with storageServer, accounts, and hosts fields for better storage infrastructure management
-
Added updateVolume method to StorageProviderVolumes interface
-
Implemented snapshot file service for advanced snapshot management
-
Added support for datastore type actions and provider enhancements
-
-
Workload and Instance Management
-
Added comprehensive workload resize v2 support with validation capabilities
-
Implemented ResizeV2WorkloadResponse with preserveVolumes flag for flexible volume handling
-
Added Host ResizeV2Facet for advanced host resizing capabilities
-
Implemented ejectAllDisksFromWorkload functionality
-
Added support for prepareCloneInstance with createLinkedClone hook
-
Added rawData field to ComputeServer model for additional metadata storage
-
-
Network Provider Evolution
-
Major refactor for CN2 network provider support with enhanced workload provisioning hooks
-
Added migration hooks for NetworkProvider.MvmProvisionFacet
-
Added IPv6 fields to Network model for dual-stack networking support
-
Enhanced NetworkRouterType with BGP option support
-
Added hasFirewall boolean to NetworkType model
-
Implemented hasSecurityGroups() method in CloudProvider class
-
-
Backup and Replication
-
Added support for multiple backup providers with comprehensive integration
-
Implemented active replica support for enhanced data protection
-
Enhanced backup provider capabilities with extended plugin integration
-
-
Compute and Provisioning
-
Added support for pre-provisioned server integration to existing instances
-
Enhanced service plan capabilities with determination flags in provision providers
-
Updated ComputeServer model with guestAgentStatus field (not QEMU exclusive)
-
Exposed computeTypeLayout and computeSet for better provisioning control
-
Added serverGroupMemberStatus for improved cluster member tracking
-
Set default visibility on VirtualImage for consistent image management
-
-
Cloud and Resource Management
-
Added pool support to cluster tabs for enhanced resource organization
-
Enhanced custom plugin info support with extended provider capabilities
-
Improved API client response memory footprint (with subsequent refinement)
-
-
Documentation and Code Quality
-
Enhanced documentation for workload resize validation
-
Improved code clarity with typo fixes and comment updates
-
-
-
1.2.12
-
Workload Resize and Validation
-
Added comprehensive support for validating workload resize operations
-
Enhanced resize workflow with proper validation hooks
-
Improved error handling during workload resize processes
-
-
Cluster and Resource Pool Management
-
Plugin cloud providers can now enable resource pool creation
-
Added finalizeLinuxComputeServer to cluster provider for enhanced Linux server management
-
Enhanced ClusterProvider with comprehensive updates and improvements
-
Improved server grouping capabilities
-
-
Storage and Volume Enhancements
-
Implemented shared storage volumes model changes
-
Enhanced volume management across cluster environments
-
Fixed field naming on ComputeServerGroup for consistency
-
-
Operation and Event Tracking
-
Introduced Operation Event Service with related models for comprehensive operation tracking
-
Added support for tracking system operations and events
-
Improved process event handling and monitoring
-
-
Console and Server Management
-
Added consoleUsername field to ComputeServer for enhanced console access
-
Added consoleKeymap field to ComputeServer for keyboard layout support
-
Improved remote console capabilities
-
-
Option Types and UI
-
Added BYTESIZE to InputType enum for option types (PCCP-4099)
-
Enhanced form field options for better user experience
-
-
Image and OS Support
-
Fixed field naming on OsTypeImage for model consistency
-
Improved OS type image handling
-
-
Performance Optimizations
-
Memory improvements in streaming QCOW2 writer for better resource utilization
-
Enhanced large file handling efficiency
-
-
Code Quality and Maintenance
-
Updated editor config with development standards
-
Improved code consistency across the project
-
Reverted redundant fields to maintain clean codebase
-
-
-
1.2.11
-
Storage and Compute Enhancements
-
Exposed StorageGroups for plugins with comprehensive service support
-
Added support for Compute Devices and enhanced compute server management
-
Added morpheusHypervisor property to ComputeServerType for better virtualization support
-
Enhanced physical interface cleanup capabilities for workload removal
-
-
Networking Improvements
-
Added NetworkRouterNAT service for improved network routing capabilities
-
Implemented FilterSubnet stub for enhanced network filtering
-
Un-deprecated networksubnets create with parent network parameter
-
Deprecated legacy filterNetworks in favor of more robust filtering methods
-
-
OS and Platform Support
-
Enhanced OS-specific hostname formatting capabilities
-
Added windowsInstallIndex to OsType model for Windows deployment support
-
Added TPM, UEFI, and SecureBoot fields to ComputeServer for enhanced security
-
-
Provider Infrastructure
-
Added daily refresh hooks for various provider types
-
Implemented flag to force resource cleanup on server delete operations
-
Enhanced VCD plugin support with additional required properties
-
Added floating IP types support for network management
-
-
Documentation and Development
-
Added comprehensive header and footer to ASCII docs and dev site
-
Improved plugin development experience with better error handling
-
-
-
1.2.10
-
Security Group and Network Enhancements
-
Added NetworkRouterNAT model for advanced network routing configurations
-
Added SecurityGroupRuleProfile and SecurityGroupRuleDestination identity projections
-
Enhanced network provider capabilities with additional VCD plugin properties
-
-
Data Management and Processing
-
Added ComputeProvisionFacet for DatastoreTypeProvider integration
-
Enhanced identity projection support in synchronous services
-
Improved model serialization with @JsonProperty annotations for "is" prefixed fields
-
-
System and Provider Improvements
-
Added system models and system provider support
-
Enhanced generic integration provider with creatable flag
-
Fixed cached i18n files handling and resolved typos
-
Added hardware CPU frequency tracking capabilities
-
-
Streaming and Storage
-
Improved streaming QCOW2 writer functionality
-
Added input stream support for StreamingQcow2Writer
-
Enhanced QCOW2 processing with streaming refactoring
-
-
-
1.2.9
-
Volume and Storage Management
-
Added additional datastore event types to support datastore changes on volumes
-
Enhanced volume lifecycle management with comprehensive event tracking
-
Improved storage provider integration capabilities
-
-
System Management
-
Added managed by property for better resource tracking
-
Introduced system flag for enhanced system resource identification
-
Fixed field name consistency across storage models
-
-
-
1.2.8
-
Process and Package Management
-
Enhanced process service APIs with comprehensive rework and synchronization support
-
Added process step types for cluster package management
-
Introduced ability to edit addon package configurations
-
Added custom form support for cluster packages with validation
-
Added ComputeServerGroup support for enhanced server grouping
-
-
Volume and Storage Operations
-
Added attach/detach volume support to datastore events
-
Enhanced volume option source capabilities
-
Improved CSI type management with delete flags for workload removal
-
-
Load Balancer and Network Integration
-
Enabled load balancer compatibility for plugin network pools
-
Enhanced network provider capabilities
-
-
Process Type Enhancements
-
Added missing ProcessTypes to enum with proper naming conventions
-
Enhanced process tracking with updatePackage support in ProcessEvent/Step definitions
-
Added request object support to addon package configuration
-
-
Cloud Provider Improvements
-
Added cloud classification to cloud provider for better categorization
-
Enhanced workload response handling with RemoveWorkloadResponse
-
Added preserveVolumes flag for better volume management during workload operations
-
-
Core Dependencies
-
Bumped Karman core version for improved functionality
-
-
-
1.2.7
-
New Process Service API
-
Introduced a new Process Service API for improved process management
-
Added support for internal process ID tracking during marshalling/unmarshalling operations
-
Enhanced process tracking capabilities between processing steps
-
-
Backup Integration Enhancements
-
Added support for backup plugin result exporting
-
Introduced custom backup integration detail views
-
Added ability to create custom backup provider tabs
-
Enhanced backup provider interface with view rendering capabilities
-
-
Network Infrastructure Improvements
-
Added complete Floating IP and Floating IP Pool services
-
Introduced NetworkResourceGroup and NetworkResourceGroupMember services
-
Added NetworkSwitch support with comprehensive service layer and models
-
Implemented ComputeServerInterfaceType services
-
-
Cluster Package Support
-
Added support for compute type packages
-
Implemented server group package functionality
-
Enhanced cluster management capabilities with package features
-
-
API Improvements
-
Added validateUsageOnDelete for NetworkTypes
-
Enhanced NetworkProvider with event subscription capabilities
-
Added support for filterServicePlans to CloudProvider
-
Enhanced StorageProviderVolumes interface with additional methods
-
Added new typed removeWorkload method and deprecated original implementation
-
Introduced suspendWorkload functionality to WorkloadProvisionProvider
-
Added ability to disable cookie usage in HTTP API Client
-
-
Bug Fixes
-
Fixed process ID tracking during marshalling/unmarshalling operations
-
Corrected fields handling in DatastoreEvent model
-
Resolved option types implementation issues in various providers
-
Fixed network provider synchronization issues
-
-
Breaking Changes
-
Deprecated existing interfaces in GenericIntegrationProvider
-
Introduced new event subscription models and interfaces (with backward compatibility)
-
Deprecated original removeWorkload implementation in favor of new typed method
-
-
-
1.2.5
-
Added support for NetworkProvider plugins creating an integration without a cloud integration
-
Storage providers were moved to the storage context: morpheus.services.storageVolume ⇒ morpheus.services.storage.volume
-
Added ability for Network Providers to add UI tabs to the network integration detail page.
-
Added connection pooling to HTTP API Client
-
-
1.2.4
-
Improvements to NetworkProvider types
-
Improvements to DatastoreTypeProviders
-
Beginnings of AffinityGroup support
-
-
1.2.3
-
Enhancements to DatastoreTypeProviders for HPE VME/MVM.
-
FileCopyService capability advancements.
-
Various Dependency Updates
-
-
1.2.0
-
OS Type Image support added to the various models to enable seeding and packaging of Os type based image associations.
-
-
1.1.9, 1.2.0
-
Added DatastoreTypeProvider plugin support for MVM. This allows custom datastore plugins to interact with hosts and LUNs throughout the provisioning vm process.
-
-
1.1.6
-
Finalized GenericIntegrationProvider and created generator
-
Finalized GuidanceRecommendationProvider and created generator
-
Finalized AnalyticsProvider and created generator
-
added endpoint for triggering agent upgrade
-
added endpoints for getting latest agent versions for current release
-
added stop/start/restart server actions
-
Handlebars and Asset reloading when a plugin is re-uploaded has been fixed.
-
Added additional endpoints for creating alarms
-
-
1.1.5
-
Deprecated
callXmlApi()method inHttpApiClient -
Improve JSON handling in HttpApiClient
-
Added
contentandcontentFormattedtoCatalogItemType -
Added
isSnapshottoBacupTypeProvider -
Added
BackupServertoResourcePermission.ResourceType -
Added
statusPercenttoVirtualImage
-
-
1.1.4
-
Added
removeInstancetoLoadBalancerProvider.
-
-
1.1.3
-
Added missing image types to
ImageTypeenum. -
Moved
importWorkloadFacettoWorkloadProvisionProvider. -
Added methods for importing workloads and retrieving storage providers for storage buckets.
-
Added location getter on synchronous virtual image service.
-
Added
vipPooltoNetworkLoadBalancerInstance. -
Includes all changes from 0.15.14
-
-
0.15.14
-
Added editable, nameEditable, and deletable flags to
StorageVolumeType. -
Introduced secure boot, TPM, and credential guard to
VirtualImage. -
Added IP normalization to
NetworkUtility.
-
-
1.1.2
-
Added support for synchronous services for backup storage.
-
Introduced methods to get bucket and bucket Karman provider for a backup.
-
Added TPM flag to
VirtualImage. -
Added new flags for default sync active state.
-
Updated forecast models.
-
Added interface method on provision providers for setting explicit descriptions on default instance types.
-
Includes all changes from 0.15.13
-
-
0.15.13
-
Added
activetoNetworkDomain. -
Introduced sync for
getCloudFileStreamUrl. -
Added
provisionRequiresResourcePoolflag to cloud types. -
Added support for filtering networks and datastores during provisioning based on the selected resource pool.
-
-
1.1.1
-
Added caching capabilities to
Qcow2InputStream. -
Introduced QCOW conversion tools/helpers for plugins.
-
Added various model maps and option types.
-
Enhanced inventory types and form field options.
-
Fixed network count and improved data query services.
-
Includes all changes from 0.15.11 and 0.15.12
-
-
0.15.12
-
Deprecated unused host types.
-
Handled
maxCoresbeing null on plan lookup. -
Allowed cloud plugins to set
canCreateNetworkson cloud types.
-
-
0.15.11
-
Made
getGlobalNetworkProxyasynchronous and synchronous. -
Added
isPluginproperty toAccountResourceType. -
Added
getGlobalNetworkProxyto the settings service.
-
-
1.0.6
-
Added
addToDatetoDateUtility. -
Introduced additional
callXmlApimethod options to matchcallJsonApimethods. -
Includes all changes from 0.15.10
-
-
0.15.10
-
Added labels to compute server.
-
Introduced appliance instance, execute schedule, and setting APIs.
-
Added
systemImagetoImageLocation. -
Added interface for scale providers.
-
Supported IaC provisioning and resource mapping.
-
Added support for workload metadata tag updates.
-
-
1.0.5
-
Added Packages to compute type layout
-
Plugin API Services
-
Includes all changes from 0.15.9
-
-
0.15.9
-
Added Packages to compute type layout
-
Updated snapshot management.
-
-
1.0.4
-
Improvements to HTTPApiClient to support Certificate Auth as well as new methods for capturing response as a stream
-
Cloud Pool management support for cloud plugins and network associations
-
Added IPv6 CIDR to NetworkPool
-
Includes all changes from 0.15.8
-
-
0.15.8
-
Improvements to HTTPApiClient to support Certificate Auth as well as new methods for capturing response as a stream
-
Cloud Pool management support for cloud plugins and network associations
-
Added IPv6 CIDR to NetworkPool
-
-
1.0.3
-
Improvements to Model serialization
-
Additional method calls to support Amazon ScaleGroups
-
Added Backup Provider templates to generator
-
Includes all changes from 0.15.7
-
-
0.15.7
-
Improvements to Model serialization
-
Additional method calls to support Amazon ScaleGroups
-
-
0.14.7
-
Added missing method in NetworkUtility
-
-
1.0.2
-
1.0 Release with proper deprecation support!
-
Moved all rxjava calls to rxjava3 from rxjava2 (NOTE: This requires all plugins to be updated for 6.3.0 of morpheus)
-
Includes all changes from 0.15.6
-
-
0.15.6
-
All Context Services now implement
MorpheusDataService -
Created SynchronousDataService equivalents for all asynchronous ones
-
Started HostProvider work for custom cluster types
-
New Task Provider format for simplification of making task plugins
-
Additional
Facetsfor injecting functionality into variousProvisionProviders -
ProvisionProvider classes split up based on type of provisioner.
WorkloadProvisionProvider,AppProvisionProvider,HostProvisionProvider, and `CloudNativeProvisionProvider.
-
-
0.15.5
-
0.15.4
-
Not released due to last minute issues
-
-
0.15.3
-
Converting More Context Services to
MorpheusDataServiceversions and deprecating old methods. -
Deprecated direct service accessors on
MorpheusContextin favor ofmorpheusContext.getAsync()for all the existing reactive services andmorpheusContext.getServices()for all the synchronous counterparts. -
Rename ComputeZonePool to CloudPool
-
Rename ComputeZoneRegion to CloudRegion
-
Rename ComputeZoneFolder to CloudFolder
-
Adding javadoc details to existing and new classes
-
Introducing
Facetinterfaces for adding additional functionality toProvisionProviderimplementations. -
Starting to rename
IdentityProjectionobjects toIdentityfor shorter naming convention. -
New Base interfaces for ProvisionProvider based on if provisioning Compute or Cloud native resources or Apps.
-
NOTE: There are breaking changes in this plugin release for cloud plugins and likely more to come as we polish for 1.0 GA
-
-
0.15.2
-
MorpheusDataService enhancements with added query methods.
-
Deprecated Service access directly on
MorpheusContextin favor of accessing thru sub classes i.e.morpheusContext.getAsync().getService(). -
Began adding non-reactive synchronous service access via
morpheusContext.getServices().getService() -
Improved javadoc for
DataQueryandDataServicemethods.
-
-
0.15.1
-
Moved most Providers new packages folder
com.morpheusdata.core.providers -
Deprecated
OptionSourceProviderin favor of newDatasetProvider-
Enables scribe export/import object reference mapping and hcl data lookup as well
-
-
Service Consistency work in the
MorpheusContext.-
Created new
MorpheusDataServiceinterface reference that allows for using dynamic db queries and object marshalling into the core/api models.
-
-
New
StorageProviderwork began for abstracting various storage providers within morpheus. -
Enhanced
NetworkProviderto supportRouterandSecurityGrouprepresentations.
-
-
0.15.0
-
Filling in more Models and Cloud representations.
-
-
0.14.4
-
Fixed an issue where the BackupProvider wasn’t marshalled to the cloud on option sources.
-
-
0.14.3
-
Filling in more Models and Cloud representations.
-
Completed Localization support. Plugins now can be fully localized in both server side, and client side rendering. Guide provided as well.
-
-
0.14.2
-
Filling in more Models and Cloud representations.
-
Added OptionType support for the
hiddenHTML Input.
-
-
0.14.1.
-
Filling in more Models and Cloud representations.
-
-
0.14.0
-
Filling in Cloud related gaps as we work to provide full cloud provider plugin support
-
F5 Load Balancer support added and full abstractions for the
LoadBalancerProvider.
-
-
0.13.4
-
Backup Plugin Support Added
-
Cloud Plugin Coverage Improved
-
DNS Plugins can now function standalone
-
HTTP ApiClient now uses CharSequence for GString compatibility
-
Improved Javadoc
-
IPAMProvider Interface removed unnecessary methods
-
Task Type Icons now use a getIcon() method on the Provider
-
Network Pool Objects added IPv6 information (more to come)
-
Context Services for Syncing additional cloud object types (such as Security Groups)
-
Various other bug fixes and improvements on the road to 1.0.0
-
Bump JVM Compatibility minimum to 1.11 (jdk 11)
-
-
0.13.1
-
Added Credential Providers support as well as significant CloudProvider refactoring (more to follow)
-
-
0.12.5
-
Task Providers now have a hasResults flag for result variable chaining.
-
-
0.12.4
-
IPAM NetworkPoolType filters for handling multiple pool types in one integration.
-
Deprecated reservePoolAddress from IPAMProvider as its no longer needed.
-
Added typeCode to the
NetworkPoolIdentityProjection. -
Added
{{nonce}}helper to handlebars tab providers for injecting javascript safely within the Content Security Policies in place.
-
-
0.12.3
-
Simplification and Polish if IPAM/DNS Interface Implementations (need Morpheus 5.4.4+).
-
Added new ReportProvider helper for easier management of db connection use
withDbConnection { connection → }.
-
-
0.12.0
-
Cloud Provider Plugin Critical Fixes (WIP).
-
Added Plugin settings.
-
-
0.11.0
-
Cloud Provider Plugin Support.
-
UI Nonce token attribute added for injecting javascript securely and css.
-
Network Provider Plugin support. Create providers for dynamically creating networks and network related objects.
-
-
0.10.0
-
Custom Report Type Providers have been added.
-
-
0.8.0
-
Overhauled DNS/IPAM Integrations, Reorganized contexts and standardized formats. \
-
Added utility classes for easier sync logic.
-
Custom reports, Cloud Providers, Server Tabs, and more.
-
Only compatible with Morpheus version
5.3.1forward.
-
-
0.7.0
-
Please note due to jcenter() EOL Don’t use 0.7.0
-
-
0.6.0
-
Primary Plugin target base version for 5.2.x Morpheus Releases
-
Getting Started
Developing plugins for Morpheus is a bit different than simply creating custom tasks or custom library items in Morpheus. It requires some programming experience. We should preface with the fact that Morpheus runs on top of Java. It is primarily written in a dynamic language called Groovy and is based on Groovy version 3.0.7 currently running within Java 11 (openjdk). The provided plugin api dependency provides a common set of interfaces a developer can implement to dynamically load functionality into the product. These developers are free to develop plugins in native Java or leverage the groovy runtime. Most examples provided are developed and sampled in groovy. It is a great dynamic language that is much less verbose and easier to understand than native java based languages. Plans for supporting additional languages are in the works. Most notably kotlin is being looked into as an alternative development platform. jRuby is also a viable language that can be used as the runtime is included with Morpheus. The plugin architecture is designed based on reactive models using rxJava2. This library follows ReactiveX models for Observer pattern based programming. One unique thing about Observer pattern programming is to remember when writing a call to an Observable the code is not executed until the final subscribe call. If this call is missing, the code will never execute.
NOTE: rxJava3 is a different project build and cannot be used yet in place of rxJava2.
The plugin pipeline leverages Gradle as the build tooling. Gradle is a cross-platform programmatic build tool that is very commonly used and is most notably also used in the android space. To begin make sure your development environment has Gradle 6.5 (at least but if using newer Gradle make sure you are familiar with the definition changes needed) as well as Java 11 (if using openjdk over 11 make sure target compatibility is set to 1.11 within your project).
NOTE: Currently Gradle 7.x is recommended as we upgrade support to Gradle 8
The morpheus-plugin-core git repository is structured such that sample plugins exist underneath the samples directory. They are part of a multi-project gradle project as can be defined in the root directories settings.gradle. This adds a bit of complexity and should be ignored when developing ones own plugin. Simply start with a blank project folder and a simple build.gradle as demonstrated below.
The structure of a Plugin is a typical "fat jar" (aka shadowJar). The plugin will include morpheus-plugin-api along with all the dependencies required to run. Though not required, it is often conventional to use the groovy programming language when developing plugins as can be seen in the samples. Morpheus is based on the groovy runtime and therefore allows full use of groovy 3.0.x.
The Structure of a project often starts with a gradle build along with a Plugin class implementation and a manifest that points to this class. This is the entrypoint for the plugin where all metadata about the plugin is defined as well as all Providers offered by the plugin are registered.
Gradle project
Create a new project with a build.gradle file. We will use the shadowjar plugin to create our "fat jar"
plugins {
id "com.bertramlabs.asset-pipeline" version "4.3.0"
id "com.github.johnrengelman.shadow" version "6.0.0"
}
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath "com.morpheusdata:morpheus-plugin-gradle:0.14.3"
}
}
apply plugin: 'com.morpheusdata.morpheus-plugin-gradle'
apply plugin: 'java'
apply plugin: 'groovy'
apply plugin: 'maven-publish'
group = 'com.example'
version = '1.0.0'
sourceCompatibility = '1.11'
targetCompatibility = '1.11'
ext.isReleaseVersion = !version.endsWith("SNAPSHOT")
repositories {
mavenCentral()
}
configurations {
provided
}
dependencies {
provided 'com.morpheusdata:morpheus-plugin-api:0.12.5' //use 0.13.4 for 5.5.x
provided 'org.codehaus.groovy:groovy-all:3.0.9'
/*
When using custom libraries, use the gradle `implementation` directive
instead of `provided`.
This will allow shadowJar to package the library into the resulting plugin and keep it
isolated within the same classloader.
*/
}
tasks.assemble.dependsOn tasks.shadowJar
To let the Morpheus plugin manager know what class to load you must specify the class in jar’s manifest:
jar {
manifest {
attributes(
'Plugin-Class': 'com.example.MyPlugin', //Reference to Plugin class
'Plugin-Version': archiveVersion.get() // Get version defined in gradle
'Morpheus-Name': 'Plugin Name',
'Morpheus-Organization': 'My Organization',
'Morpheus-Code': 'plugin-code',
'Morpheus-Description': 'My Plugin Description',
'Morpheus-Logo': 'assets/myplugin.svg',
'Morpheus-Logo-Dark': 'assets/myplugin-dark.svg',
'Morpheus-Labels': 'Plugin, Stuff',
'Morpheus-Repo': 'https://github.com/myorg/myrepo',
'Morpheus-Min-Appliance-Version': "5.5.2"
)
}
}
When writing plugin code, it is important to note a typical groovy/java project folder structure
./
.gitignore
build.gradle
src/main/groovy/
src/main/resources/renderer/hbs/
src/main/resources/i18n
src/main/resources/scribe
src/main/resources/packages
src/test/groovy/
src/assets/images/
src/assets/javascript/
src/assets/stylesheets/
Be sure to configure your .gitignore file to ignore the build/ directory which appears after performing your first build.
Most of the folder structure can be self-explanatory if familiar with groovy/java. Project packages live within src/main/groovy and contain source files ending in .groovy. View resources are stored in the src/main/resources subfolder and vary depending on the view renderer of choice. While static assets like icons or custom javascript live within the src/assets folder. This is handled by the Asset Pipeline plugin. View rendering and static assets will be covered in more detail later.
Building a project is as simple as calling
./gradlew shadowJar
NOTE: If the gradle wrapper does not yet exist, simply run gradle wrapper within the root of the project to generate the wrapper.
The resulting jar will exist, by default, in the build/libs directory of the project.
Plugin Class
Following the example above, create your plugin class under src/main/java/com/example/MyPlugin.java
Your plugin must extend the com.morpheus.core.Plugin class:
import com.morpheus.core.Plugin;
class MyPlugin extends Plugin {
@Override
void initialize() {
this.setName("My Custom Tabs Plugin");
CustomTabProvider tabProvider = new CustomTabProvider(this, morpheus);
this.registerProvider(tabProvider);
}
}
Here we see a basic plugin that initializes some metadata (the Plugin Name) and registers a custom tab provider.
Registering Providers
A plugin may contain any number of Plugin Providers. A Plugin Provider contains the functionality of your plugin such as a Custom Tab, IPAM, Backup Provider, etc.
There are provided classes such as TaskProvider, ProvisionProvider, ApprovalProvider, ReportProvider, InstanceTabProvider, ServerTabProvider, and others to get you started building your provider. For example the InstanceTabProvider provides a renderer, show method, and security checking method to make it easy to build your own custom tab on the instance page.
Providers are registered in your plugin during the initialize() method call within the Plugin class as can be seen in the MyPlugin.java sample seen above.
Settings
As of 0.8.0 Plugins can now have settings that can be applied globally after installing a plugin. Some users may use this to configure an integration to a third party service like Datadog, or affect how providers may behave based on some setting. These can be set after uploading a plugin in Administration → Integrations → Plugins.
Settings, much like any other form aspect of a plugin, can take advantage of OptionType and OptionProvider entities to configure how the options are presented to the user and what options are available to choose from.
class MyPlugin extends Plugin {
/**
* Returns a list of {@link OptionType} settings for this plugin.
* @return this list of settings
*/
public List<OptionType> getSettings() { return this.settings; }
}
Implementing the above getter method allows one to specify the form option-types of settings that could be saved by the user.
Fetching setting values for use in a provider can easily be accomplished via the morpheusContext. There is a method that returns a JSON String of the setting values that is up to the plugin developer to deserialize called morpheusContext.getSettings(Plugin plugin). Simply pass the plugin class instance to it and an rxJava Single<String> is returned.
String pluginSettings = morpheus.getSettings(this.plugin).blockingGet()
def pluginDeserialized = new JsonSlurper().parseText(pluginSettings)
Contexts / Services
The Morpheus context allows you to interact with, query and save data with Morpheus. It is organized into several sub service classes to perform operations that may involve database calls or calling common methods within the Morpheus core that are not available directly. Some calls may be as simple as listById or save. But they can also be as complex as executeSshCommand or executeWinrmCommand. All calls within a Morpheus Context implement RxJava2 conventions. For details on how to program in Reactive syntax and RxJava concepts please see the documentation site http://reactivex.io/.
When interacting with various subcontexts it is helpful to know there is a common guideline on method names involving database calls. These common method names include:
-
listIdentityProjections -
listById -
listBy* -
get -
findBy* -
remove -
create -
save
There are, of course, exceptions and non ORM related methods that also may exist in certain services that provide common helper methods. For a full listing of the various service classes please check the API Docs and look at the MorpheusContext class. There are several in line with general ORM calls as well as some related to certain actions.
Models
When interacting with Morpheus you must use the models provided in com.morpheusdata.model.*. This allows for the context to be strongly typed. These Models have a few conventions that you might like to know about. Firstly, models that are often synced from a cloud provider or integration of any kind often inherit from their base class known as an IdentityProjection which also inherits from MorpheusModel.
For Example: the NetworkDomain model inherits NetworkDomainIdentityProjection which in turn inherits MorpheusModel.
The IdentityProjection contains a subset of properties that are typically used to match the object with its equivalent on the other side of an Api implementation. This is typically persisted in the externalId property of the object. Some objects also leverage uniqueId as well.
It is also recommended to use the getter and setter methods on the Models (which should be strictly required in the near future) so as to ensure the dirtyProperties map is updated appropriately. This will be used in future distributed worker installations to reduce bandwidth in transmission of data updates back to Morpheus.
Data Services
As the Morpheus Plugin API has progressed a new concept was created in 0.15.x. It was determined a consistent means for querying Morpheus data objects needed to be made across the board in all of the MorpheusContext accessible services. Previously, there were only a fixed set of options per Model that one might have wanted to query. But with the development of the DataService model and the new DataQuery object, a lot more power is available to the developer.
These services are now commonly used in various sync related activities with SyncTask as well as can be used with custom reporting or the new DataSetProvider concepts.
public interface MorpheusDataService<M extends MorpheusModel, I extends MorpheusModel> {
Observable<M> listById(List<Long> ids);
Observable<M> list(DataQuery query);
Observable<Map> listOptions(DataQuery query);
Maybe<M> find(DataQuery query);
Single<DataQueryResult> search(DataQuery query);
}
Above you will see a common set of methods for querying data. The important part of these methods is to note that they all take a DataQuery object. This object allows you to scope a query to a User or an Account within morpheus as well as pass in complex query operators.
For Example:
def usageClouds = morpheusContext.async.cloud.list(
new DataQuery().withFilters(
new DataOrFilter(
new DataFilter("externalId","in",usageAccountIds),
new DataFilter("linkedAccountId","in",usageAccountIds)
)
)
).toList().blockingGet()
The above query looks for all clouds that contain a set of usage account ids via either the externalId property or the linkedAccountId property.
As can be seen above, queries can be nested with DataOrFilter or DataAndFilter combinations to build complex queries. More details of this also exist on the DataQuery java doc.
NOTE: The DataService also provides save, create,remove methods of a consistent nature. Please refer to the APIDoc / javadoc for descriptions on how to use these methods.
In the past, when making custom reports, it was common to use the direct database connection and query the morpheus data set directly. This carried with it some risks as these tables are not strictly documented and some data is in an encrypted state. By utilizing data services ( where possible ), however, the object models are marshalled into a properly documented format and fields that may have been previously inaccessible, due to encryption, are now available for use.
Synchronous Data Services
In an effort to make plugin development easier, it was decided that a counterpart service would be made for non rxjava / reactive calls. We call this the SynchronousDataService. This service provides the exact same methods as the MorpheusDataService, however, it does not require an Observable subscription or blockingGet().
These services can be accessed via the contexts morpheusContext.getServices() directive as opposed to morpheusContext.getAsync(). They are useful for things that you know are going to block anyway, such as perhaps a UI Page render.
It is still recommended, when performing sync operations of a cloud, or generating a custom report to utilize the asynchronous counterparts for best performance.
NOTE: As of 0.15.3: Not All Async services have been converted yet and several may still be missing.
Syncing Data
Many of the Morpheus Provider types have a method to periodically refresh and sync data into Morpheus. This can be useful for representing core concepts or doing brownfield discovery for things like ComputeServer or NetworkPool objects. Also, it may be useful for syncing in user presented options in a dropdown or typeahead during a particular operation (Generic Data for this we can store n ReferenceData).
Over the course of several years, Morpheus has developed an optimal way to efficiently sync data from a remote endpoint into the appliance. This has to take into account network bandwidth, memory , cpu, and load on the target api. Morpheus also created a helper class for making sync operations simple and consistent called the SyncTask (javadoc available).
Below is an example that syncs available amazon regions in the AWS Cloud to the CloudRegion table.
Observable<CloudRegionIdentity> domainRecords = morpheusContext.async.cloud.region.listIdentityProjections(cloud.id)
SyncTask<CloudRegionIdentity, Region, CloudRegion> syncTask = new SyncTask<>(domainRecords, regionResults.regionList as Collection<Region>)
syncTask.addMatchFunction { CloudRegionIdentity domainObject, Region data ->
domainObject.externalId == data.getRegionName()
}.onDelete { removeItems ->
removeMissingRegions(removeItems)
}.onUpdate { List<SyncTask.UpdateItem<CloudRegion, Region>> updateItems ->
updateRegions(updateItems)
}.onAdd { itemsToAdd ->
addMissingRegions(itemsToAdd, this.@cloud.account)
}.withLoadObjectDetailsFromFinder { List<SyncTask.UpdateItemDto<CloudRegionIdentity, Region>> updateItems ->
morpheusContext.async.cloud.region.listById(updateItems.collect { it.existingItem.id } as List<Long>)
}.start()
NOTE: The above example does not show that the api query to get all regions was performed above.
A SyncTask is capable of taking the Identity objects of a class (small form of a Model that contains just the important identification fields used for matching) and comparing them against the api results list. This is done via the addMatchFunction as seen above.
NOTE: It is possible to have a chain of match functions as secondary fallbacks by simply adding another to the chain.
Objects that do not have a match from the api are sent in batches to the onAdd for creation of Morpheus Model objects and consequently the inverse is true for models that are not found within the api results as these are sent to onDelete.
Objects that do match are a bit different. Firstly, in order, to check the object for changes it must first be fully loaded from its Identity. This can be seen in the withLoadObjectDetailsFromFinder call. This translates those Identity classes into their fully loaded objects in batches of 50 at a time (not seen as this is magically done by the SyncTask). Once these objects are fully loaded they are passed down into the onUpdate where they can be compared.
Although not seen above, it is best to bulk create or bulk save morpheus changes in these chunked add and update methods. This is done using the MorpheusDataService.bulkCreate or MorpheusDataService.bulkSave implementations on the context.
Sync methods such as this exist in several provider types beyond just clouds. IPAM Providers, DNS Providers, backups and others all contain endpoints where these sync operations can be performed.
HTTP Routing
A plugin may register an endpoint or endpoints to respond with html or json. To register your routes in your plugin you must implement the PluginController class.
Models
Incoming requests come with a ViewModel object populated with the http request & response details.
Example
class MyPluginController implements PluginController {
List<Route> getRoutes() {
[
Route.build("/myPrefix/example", "html", Permission.build("admin", "full")),
Route.build("/reverseTask/json", "json", Permission.build("admin", "full"))
]
}
def html(ViewModel<String> model) {
return HTMLResponse.success("Some Text")
}
def json(ViewModel<Map> model) {
Map simpleMap = [serverid: "abc-123", other: model.object.someData]
return JsonResponse.of(simpleMap)
}
}
Route provides a builder to allow your plugin to easily build a route with permissions. It takes the url, the method in this class to call, and a list of permissions which can be built with the Permission builder.
The route can either return:
-
HTMLResponse- simple text, or a full rendered view. -
JsonResponse- an object rendered as json.
After creating a PluginController, register it in your plugin like so:
@Override
void initialize() {
this.setName("My Custom Task Plugin");
CustomTaskProvider taskProvider = new CustomTaskProvider(this, morpheusContext);
this.pluginProviders.put(taskProvider.providerCode, taskProvider);
this.controllers.add(new MyPluginController());
}
HTTP API Client
The morpheus-plugin-api library comes with a utility for facilitating quick and easy API calls to external integrations. This is called teh HttpApiClient and replaces the use of the RestApiUtil.
The developer is, of course, able to utilize any HTTP or SDK/API client they choose within the java ecosystem if so desired. However, this client provides some common functions such as SSL Validation, Throttling, Keep-Alive, Etc. and is based on the Apache HTTP Client.
|
Tip
|
Reuse the same client instance when dealing with periodic refresh methods related to caching for best performance. Be mindful that throttling might be necessary to slow down the calls to the service. |
Example
Below is an example API Call taken from the Infoblox plugin.
import com.morpheusdata.core.util.HttpApiClient
HttpApiClient infobloxClient = new HttpApiClient()
try {
def results = infobloxClient.callJsonApi(serviceUrl, apiPath, poolServer.serviceUsername, poolServer.servicePassword, new HttpApiClient.RequestOptions(headers:['Content-Type':'application/json'],
queryParams:pageQuery, ignoreSSL: poolServer.ignoreSsl), 'GET')
} catch(e) {
log.error("verifyPoolServer error: ${e}", e)
} finally {
infobloxClient.shutdownClient()
}
return rtn
There are several methods to call the api and automatically handle certain payload formats such as callJsonApi, callXmlApi, or plain callApi.
Options can be passed relating to the request via the HttpApiClient.RequestOptions object. Please refer to the java api doc for available options.
|
Tip
|
Remember to always shutdown the client after it is used to clean up the connection manager in a try{} finally{} type block
|
Views
Plugins may render html sections such as adding a tab to different areas of Morpheus which you can populate with your own content. By default a Handlebars template provider is provided by the Plugin Manager. If you wish to use your own template engine you may implement com.morpheusdata.views.Renderer interface.
Rendering a view
Views are stored in your plugin at src/main/resources/renderer/<plugin scope>/<your view>.hbs. Many plugin providers provide a convention to store views that will be rendered.
If you wish to render and return html manually you can call the renderer directly:
getRenderer().renderTemplate("prefix/someview", model);
Do not provide the suffix (.hbs) - you may also pass a com.morpheusdata.views.ViewModel into the view to use when rendering the html.
Asset helper
The template engine can also process static assets such as images, css, and javascript for you. Place your assets under src/assets/{plugin-code}.
To include them in the plugin template you can use the {{ asset }} handlebars helper as so:
<script src="{{asset "/instance-tab.js"}}"></script>
<img src="{{asset "/some/image.png"}}" />
Localization
Morpheus Plugins support using i18n localization properties. These are string maps for representing various sections of text in your plugin (or in morpheus itself) by a localized way. Creating a messages.properties file in the directory src/main/resources/i18n will allow you to set some key value maps.
For Example messages.properties:
com.morpheusdata.label.hello=Hello
or for Spanish (ES) messages_es.properties:
com.morpheusdata.label.hello=Hola
These can be leveraged in the handlebarsRenderer for server side rendering via the {{i18n}} helper like so:
<strong>{{i18n 'com.morpheusdata.label.hello'}}</strong>
Or they can also be used in javascript rendering (like in dashboard widgets or other areas):
var helloString = $L({code: 'com.morpheusdata.label.hello', default: 'Hello'});
If a matching string exists for the current browser locale, it will be used, if not it will fall back to the default messages.properties file.
Localizing the Entire Application
Plugin localization files are global. This means, if a language pack did not exist for the entire Morpheus Appliance for a specific locale and a developer/partner wanted to make one as a plugin, they could. The plugins properties are automatically checked system-wide. It is even possible to override single labels in the morpheus appliance if so desired.
For information on localizing Morpheus, please refer to the Morpheus Crowdin Localization Plugin
DataSet Providers
A DataSetProvider is a server side call to load a dynamic data for custom form inputs, as well as data references with scribe templates. A dataset can feed data to dropdown lists, multiselect components, and typeahead components. Datasets can be very useful when designing task types that have custom options or even custom catalog item layouts. In the past, this type of data was provided with an OptionSourceProvider which contained multiple definitions in a single provider, but had its own limitations and did not provide sufficient documentation on the data provided. The DataSetProvider improved dynamic data definitions by: isolating one dataset per provider, including associative information so that it can be used for export/import scribe functionality, and including documentation on the dataset within the dataset definition.
Getting Started
Create an implementation of a DataSetProvider or with some convenience methods an AbstractDataSetProvider.
When defining an OptionType set the optionSource property to the getKey() defined in the dataset provider. Avoid naming conflicts with other plugins with unique method name or isolate the dataset by using a namespaces. A namespace is set on the namespace property of the DatasetInfo in getDataset(). Alternatively, the getNamespace() method can be implemented or overridden. When a dataset is namespaced the optionSourceType of an OptionType or OptionSource should correlate to the dataset’s namespace.
|
Tip
|
Use a unique namespace for your DataSetProvider to isolate from other plugins. |
OptionType ot4 = new OptionType(
name: 'Region',
code: 'google-plugin-region',
fieldName: 'googleRegionId',
optionSourceType: 'google', // Note: this references the dataset namespace.
optionSource: 'googlePluginRegions', // Note: this references the dataset key.
displayOrder: 3,
fieldLabel: 'Region',
required: true,
inputType: OptionType.InputType.SELECT,
dependsOn: 'google-plugin-project-id',
fieldContext: 'config'
)
DatasetProvider examples can be found in the example plugin repository on GitHub.
Option Sources
NOTE These have been replaced by DataSet Providers as of 0.15.x.
DEPRECATED: An Option source is a server side call to load a dynamic dataset for custom form inputs. These normally feed dropdown lists, multiselect components, and typeahead components. These can be very useful when designing task types that have custom options or even custom catalog item layouts.
Getting Started
To get started, simply create an implementation of an OptionSourceProvider. The primary implementation method is called getMethodNames and just returns a list of methods on the class that are accessed via API as datasets.
class GoogleOptionSourceProvider implements OptionSourceProvider {
@Override
List<String> getMethodNames() {
return new ArrayList<String>(['googlePluginProjects', 'googlePluginRegions', 'googlePluginZonePools', 'googlePluginMtu'])
}
def googlePluginProjects(args) {
Map authConfig = getAuthConfig(args)
def projectResults = []
if(authConfig.clientEmail && authConfig.privateKey) {
def listResults = GoogleApiService.listProjects(authConfig)
if(listResults.success) {
projectResults = listResults.projects?.collect { [name: it.name, value: it.projectId] }
projectResults = projectResults.sort { a, b -> a.name?.toLowerCase() <=> b.name?.toLowerCase() }
}
}
projectResults
}
}
Above is an example option source registered for the Google Cloud Plugin. NOTE: Every method must have a singular input argument of type Object (which is the default in Groovy). In reality, it is a Map containing passed in params from the api call that can be referenced as well as the current user object. It is also worth noting that the return type expected is a List<Map<String,String>> whereby the properties on the map are of keys name,value.
Now, when defining your OptionType simply set your optionSource property to the field name defined in your provider. It is important not to conflict names with other plugins so please try to use a unique method name.
|
Tip
|
Use a Unique Method name that will not interfere with other plugins the user may load. |
OptionType ot4 = new OptionType(
name: 'Region',
code: 'google-plugin-region',
fieldName: 'googleRegionId',
optionSource: 'googlePluginRegions', //Note the Option Source Defined here.
displayOrder: 3,
fieldLabel: 'Region',
required: true,
inputType: OptionType.InputType.SELECT,
dependsOn: 'google-plugin-project-id',
fieldContext: 'config'
)
Credential Provider Inputs
During the 5.4.x and 5.5.x release of Morpheus. Credentials were introduced. It became possible to store integration credentials externally or decoupled from a specific integration. This is great for service accounts! As a result a new option type was created to allow them to be used for custom plugin integrations.
Getting Started
Most of the time, this setup will be used when defining available OptionTypes in the various Provider types that have them. For example, an IPAMProvider as a method for getIntegrationOptionTypes(). The idea is to also support local credential inputs like the simple serviceUsername and servicePassword fields. To do this simply add the flag localCredential:true to those fields so the system knows, when using local credentials, to show those fields. An example of a credential provider being used can be seen here.
class InfobloxProvider implements IPAMProvider, DNSProvider {
@Override
List<OptionType> getIntegrationOptionTypes() {
return [
new OptionType(code: 'infoblox.serviceUrl', name: 'Service URL', inputType: OptionType.InputType.TEXT, fieldName: 'serviceUrl', fieldLabel: 'API Url', fieldContext: 'domain', placeHolder: 'https://x.x.x.x/wapi/v2.2.1', helpBlock: 'Warning! Using HTTP URLS are insecure and not recommended.', displayOrder: 0, required:true),
new OptionType(code: 'infoblox.credentials', name: 'Credentials', inputType: OptionType.InputType.CREDENTIAL, fieldName: 'type', fieldLabel: 'Credentials', fieldContext: 'credential', required: true, displayOrder: 1, defaultValue: 'local',optionSource: 'credentials',config: '{"credentialTypes":["username-password"]}'),
new OptionType(code: 'infoblox.serviceUsername', name: 'Service Username', inputType: OptionType.InputType.TEXT, fieldName: 'serviceUsername', fieldLabel: 'Username', fieldContext: 'domain', displayOrder: 2,localCredential: true),
new OptionType(code: 'infoblox.servicePassword', name: 'Service Password', inputType: OptionType.InputType.PASSWORD, fieldName: 'servicePassword', fieldLabel: 'Password', fieldContext: 'domain', displayOrder: 3,localCredential: true),
...
}
Note the infoblox.credentials option type and its use of the type OptionType.InputType.CREDENTIAL. The optionSource is critical as well as the list of possible credential types seen in the config block.
There are other credential types available that will be populated in this documentation later.
Using Credentials in your plugin.
When making remote calls to the integration code it is important to reference the credential data correctly. For example, it is no longer simply a matter of referencing poolServer.serviceUsername or poolServer.servicePassword. Instead it is important to check the credentialData map on the AccountIntegration, Cloud, or NetworkPoolServer.
results = client.callApi(serviceUrl, apiPath, poolServer.credentialData?.username ?: poolServer.serviceUsername, poolServer.credentialData?.password ?: poolServer.servicePassword, new HttpApiClient.RequestOptions(headers:['Content-Type':'application/json'], ignoreSSL: poolServer.ignoreSsl,body:body), 'POST')
An example can be seen above where the credentialData is first checked.
Testing
We recommend Spock for easily testing your plugin. The interfaces can easily be mocked and stubbed to allow you to test your integrations without a running Morpheus instance.
Example
class InfobloxProviderSpec extends Specification {
@Shared MorpheusContext context
@Shared InfobloxPlugin plugin
@Shared InfobloxAPI infobloxAPI
@Shared MorpheusNetworkContext networkContext
@Subject @Shared InfobloxProvider provider
void setup() {
// Create a Mocks of the Morepohus contexts you will use
context = Mock(MorpheusContextImpl)
networkContext = Mock(MorpheusNetworkContext)
context.getNetwork() >> networkContext
plugin = Mock(InfobloxPlugin)
infobloxAPI = Mock(InfobloxAPI)
// Create the actual provider to unit test
provider = new InfobloxProvider(plugin, context, infobloxAPI)
}
void "listNetworks"() {
given: "A pool server"
def poolServer = new NetworkPoolServer(apiPort: 8080, serviceUrl: "http://localhost")
// Here we are stubbing the actual API call to infoblox, but we could create a integration test by actually providing the real infoblox API class instead of a mock.
infobloxAPI.callApi(_, _, _, _, _, _) >> new ServiceResponse(success: true, errors: null , content:'{"foo": 1}')
when: "We list the networks"
def response = provider.listNetworks(poolServer, [doPaging: false])
then: "We get a response"
response.size() == 1
}
}
As you can see, implementing unit and integration testing for your plugins can be done easily with Spock. Of course any other JVM unit testing framework should work as well.
Seeding data during plugin installation
If you need to ensure that certain data (e.g. layouts, plans, etc.) are added to the database when your plugin is installed, you can use this process to "seed" that data. Morpheus consumes and produces HCL formatted data. The same principles that you can use here for plugins also apply to Morpheus packages.
For plugins, you will need to create a folder called scribe in your your plugin’s resources folder (src/main/resources/scribe). Then add scribe files to that folder. We recommend descriptive names like Ubuntu22Layouts.scribe. The scribe files should be modeled off of plugin database model classes. The resource type should be the model name in dash-case and the resource should have a unique identifier (the code field for most models). All new scribe files and updates to existing those files will be processed when your plugin is installed or reloaded.
Resources can reference other resources in the same .scribe file, other resources from different .scribe files, resources from other plugins, and even existing resources in Morpheus. The references are made by specifying the resource type and a unique identifier, often the code. Just make sure that if your plugin expects resources from other plugins or packages, those plugins or packages are loaded first.
Examples
This is an option type. Note that the optionSource field is a reference to a DatasetProvider.
resource "option-type" "demo-option" {
name = "Demo Option"
code = "demo-option"
fieldName = "demoOption"
fieldContext = "config"
fieldLabel = "Demo Option"
type = "select"
displayOrder = 10
required = true
optionSource = "demoOptionSource"
}
This is an instance type that references the option type above. They could be in the same .scribe file, or different files. However, if they come from different plugins, make sure that the plugin that provides the option type is loaded first.
resource "instance-type" "demo-instance" {
name = "Demo Instance"
code = "demo-instance"
description = "Spin up any VM on our Demo infrastructure."
environmentPrefix = "DEMO"
category = "cloud"
active = true
enabled = true
versions = ["1.0"]
optionTypes = [
option-type.demo-option
]
provisionTypeDefault = true
pluginIconPath = "demo.svg"
pluginIconHidpiPath= "demo.svg"
pluginIconDarkPath = "demo-dark.svg"
pluginIconDarkHidpiPath = "demo-dark.svg"
}
This is an example instance type layout that references the built-in Morpheus Ubuntu instance type. The provisionType type here references a ProvisionProvider from a Cloud Plugin.
resource "instance-type-layout" "demo-ubuntu-22" {
code = "demo-ubuntu-22"
name = "Demo VM"
sortOrder = 22
instanceVersion = "22"
description = "This will provision a single vm"
instanceType {
code = "ubuntu"
}
serverCount = 1
hasAutoScale = true
portCount = 1
serverType = "vm"
enabled = true
creatable = true
supportsConvertToManaged = true
provisionType = "demo-provision-provider"
}
Process Service
Morpheus plugins support recording processes and their steps (AKA ProcessEvents or events). Processes represent a history of actions taken in relation to different models within Morpheus (e.g., Workload, Cluster, Server). Each Process and ProcessEvent has a ProcessStepType associated with it that describes the process or an individual event within the Process.
|
Note
|
At this time in plugins, Processes can only be created in relation to a Workload.
|
|
Tip
|
The morpheus-omega-plugin provides various examples of interacting with the process service. |
Working with Existing Processes
For some APIs, Morpheus Core creates and associates a Process. The plugin can add events to the existing process to show steps taken in the given context. For example, the WorkloadProvisionProvider APIs have a request object that provides a process during the different stages of provisioning (e.g., prepareWorkload, runWorkload, etc.). With this process available, the plugin can add steps showing an action it has taken during the provisioning flow:
ServiceResponse<ProvisionResponse> runWorkload(Workload workload, WorkloadRequest workloadRequest, Map opts) {
context.services.process.startProcessStep(workloadRequest.process, new ProcessEvent(stepType: ProcessStepType.EXECUTE_ACTION), "running")
// do some action
context.services.process.endProcessStep(workloadRequest.process, MorpheusProcessService.STATUS_COMPLETE, "action output", true)
return ServiceResponse.success(new ProvisionResponse(workload))
}
Creating a New Process
Since plugins have quite a bit of freedom in how they interact with Morpheus, there are scenario when you’re outside
a particular core flow in Morpheus, such as a custom HTTP endpoint. In this circumstance, you first need to create
a new Process associated with a model (e.g., Workload) and then add steps if needed:
class ExampleController implements PluginController {
// Note: Everything else in the class omitted for brevity
def json(ViewModel<Map> model) {
Process process = morpheusContext.services.process.startProcess(workload, ProcessStepType.EXECUTE_ACTION, null, 'timerCategory', 'eventCategory')
morpheusContext.services.process.startProcessStep(workloadRequest.process, new ProcessEvent(stepType: ProcessStepType.EXECUTE_ACTION), "running")
// do some action
morpheusContext.services.process.endProcessStep(workloadRequest.process, MorpheusProcessService.STATUS_COMPLETE, "action output", true)
morpheusContext.services.process.endProcess(process, MorpheusProcessService.STATUS_COMPLETE, 'output')
return JsonResponse.of(model.object)
}
}
Updating Steps of a Process
When steps within a process are long-running, it can be helpful to provide feedback to the user. Plugins can update a step’s output and status to reflect the current state of a step:
context.services.process.startProcessStep(workloadRequest.process, new ProcessEvent(stepType: CUSTOM_STEP_TYPE), "")
def totalTime = 60000
def increment = 10000
for (int i = 0; i < totalTime; i += increment) {
def update = new ProcessStepUpdate(
status: "running (${i}ms)",
output: "output@${i}ms\n",
)
context.services.process.updateProcessStep(workloadRequest.process, CUSTOM_STEP_TYPE, update, true)
sleep(increment)
}
context.services.process.endProcessStep(workloadRequest.process, MorpheusProcessService.STATUS_COMPLETE, "final", true)
Defining Custom ProcessStepTypes
Plugins can seed in their own custom ProcessStepTypes via scribe files. Here’s an example:
resource "process-step-type" "example-step" {
code = "example.custom-step"
name = "Example Step"
description = "Your description goes here"
}
The name will be used in the History representation of the associated Process or ProcessEvent.
In the plugin, this seeded ProcessStepType can be referenced by code like so:
ProcessStepType EXAMPLE_STEP_TYPE = ProcessStepType.forCode("example.custom-step")
Examples
Approvals plugin
Integrate Morpheus with your own ITSM solution. See a full example.
Setup
This plugin will enable you to create configuration for several aspects of Approvals within Morpheus.
Integration
A new Integration Type will be created when this plugin is installed.
You are able to customize the OptionType for the new Integration using the ApprovalProvider.integrationOptionTypes method.
These OptionType will be visible when creating the new Integration in the Morpheus UI (Administration → Integrations).
Policies
Policies (Administration → Policies in the Morpheus UI) define the conditions in which approval is required for provisioning.
Custom OptionType can be defined for Policy creation by implementing the ApprovalProvider.policyOptionTypes method.
Create Approval
ApprovalProvider.createApprovalRequest is called after a Provision Request is created.
Here is where you can send the request to your ITSM.
Each Request will have one or more RequestReference for each resource associated with the provision request.
|
Important
|
It is important that you specify an externalId in the RequestResponse and each RequestReference so that Morpheus can track the approval status.
|
The integrationOptionTypes you specified are available in the method argument Policy.configMap
String itsmEndpoint = accountIntegration.configMap?.cm?.plugin?."itsm-endpoint"
and the policyOptionTypes you specified are available in the method argument AccountIntegration.configMap.
String myPolicyConfigValue = policy.configMap?."my-policy-config-option"
Monitor Approval
At a regular interval, Morpheus checks for Request approvals. In the ApprovalProvider.monitor method
define your logic for retrieving a list of approval requests in your ITSM solution.
Approval RequestReference should be returned with one of the following ApprovalStatus:
-
requesting -
requested -
error -
approved -
rejected -
cancelled
Integration Logo
A custom logo can be used in the Morpheus UI by placing an image at src/assets/images/{plugin-code}.png.
Recommended file size is 180 x 60 px.
Backups Plugin
Backups are a big part of self-service workload management. Allowing a customer to back up their application and especially restore it on the fly are critical to creating a complete self-service solution. This is why Morpheus integrates with several best-in-class backup solutions like Veeam, Rubrik, Commvault, and more. Morpheus even provides its own basic backup providers for those customers still looking for a final solution.
Plugin Setup
First we need to register the backup provider in the plugin
class MyPlugin extends Plugin {
@Override
String getCode() {
return 'my-plugin'
}
@Override
String getName() {
return 'My Plugin'
}
@Override
void initialize() {
MyPluginBackupProvider backupProvider = new MyPluginBackupProvider(this, morpheus)
registerProvider(backupProvider)
}
Creating a BackupPlugin involves registration of 2 types of providers: Backup Provider and Backup Type Provider.
Backup Provider
The primary entry point into the plugin is the Backup Provider. The backup provider will handle all the high level operations within morpheus including creating and syncing the integration.
The example below is Backup provider is a single Backup Provider with multiple Backup Type Providers.
class MyPluginBackupProvider extends AbstractBackupProvider {
MyPluginBackupProvider(Plugin plugin, MorpheusContext morpheusContext) {
super(plugin, morpheusContext)
MyVmwareBackupProvider vmwareBackupProvider = new MyVmwareBackupProvider(plugin, morpheus)
plugin.registerProvider(vmwareBackupProvider)
addScopedProvider(vmwareBackupProvider, "vmware", null)
MyHypervBackupProvider hypervBackupProvider = new MyHypervBackupProvider(plugin, morpheus)
plugin.registerProvider(hypervBackupProvider)
addScopedProvider(hypervBackupProvider, "hyperv", null)
}
@Override
String getCode() {
return 'my-backup-provider'
}
@Override
String getName() {
return 'My Backup Provider'
}
@Override
Icon getIcon() {
return new Icon(path:"icon.svg", darkPath: "icon-dark.svg")
}
...
}
NOTE: The AbstractBackupProvider is convenient to avoid implementing standard methods and settings defined in the BackupProvider interface that are not used as often.
A Backup Type Provider MyVmwareBackupProvider is created and scoped by a provision type code. Adding a scoped provider will associate the provider to the provision type defined by the scoped provider. In the example a backup provider is defined for provisioning VMware instances.
Backup Type Provider
All operations regarding the instance of a backup, including the creation and restoring of a backup, are handled by the BackupTypeProvider. A Backup Provider also can have many Backup Type Providers. For example some backup solutions can back up VMs on many cloud types and they each may have different relevant APIs.
class MyVmwareBackupProvider extends AbstractBackupTypeProvider {
MyVmwareBackupProvider(Plugin plugin, MorpheusContext context) {
super(plugin, context)
}
@Override
String getCode() {
return "my-vmware-backup-provider"
}
@Override
String getName() {
return "MY VMware backup provider"
}
@Override
Collection<OptionType> getOptionTypes() {
return new ArrayList()
}
@Override
BackupExecutionProvider getExecutionProvider() {
return new SnapshotExecutionProvider()
}
@Override
BackupRestoreProvider getRestoreProvider() {
return new SnapshotRestoreProvider()
}
...
}
The implementation from this point is flexible. The developer can choose to implement the execute and restore functionality directly in the backup type provider or in separate providers. The execute and restore providers are simply a way to organize the provider for clarity. There are various service interfaces provided for implementing the backup and restore behavior: BackupExecutionProvider, BackupRestoreProvider.
See the Rubrik Plugin for a full example implementation including execution and restore providers.
Syncing Provider Data
Like most other integrations, a periodic refresh method is called to sync in any necessary data the integration might need. It is recommended to use a SyncTask in these refresh methods which are optimized to handle blocking vs non-blocking thread scheduling.
@Slf4j
class MyPluginBackupProvider extends AbstractBackupProvider {
/... .../
@Override
ServiceResponse refresh(BackupProvider backupProvider) {
ServiceResponse rtn = ServiceResponse.prepare()
try {
new BackupSyncTask().execute()
} catch(Exception e) {
log.error("error refreshing backup provider {}::{}: {}", plugin.name, this.name, e)
}
return rtn
}
/... .../
}
Custom Views
A backup provider can supply custom UI tabs or override the entire backup integration detail view. Override the renderTemplate() method to override the integration view. When using the default integration view additional tabs can be added by extending the AbstractBackupIntegrationTabProvider class.
The example below shows how to add a custom tab to the default backup provider detail view.
class MyBackupTabProvider extends AbstractBackupIntegrationTabProvider {
/... .../
@Override
HTMLResponse renderTemplate(BackupProvider backupProvider) {
ViewModel<BackupProvider> model = new ViewModel<>()
model.object = backupProvider
return getRenderer().renderTemplate("hbs/myTabView", model)
}
@Override
Boolean show(BackupProvider backupProvider, User user, Account account) {
// only show this tab for providers that match this plugin's backup provider
return backupProvider.type.code == MyBackupProvider.PROVIDER_CODE
}
}
Override the enter view by implementing the renter template method in your backup provider class:
class MyBackupProvider extends AbstractBackupProvider {
private HandlebarsRenderer renderer
/... .../
@Override
HTMLResponse renderTemplate(com.morpheusdata.model.BackupProvider backupProvider) {
ViewModel<com.morpheusdata.model.BackupProvider> model = new ViewModel<>()
model.object = backupProvider
return getRenderer().renderTemplate("hbs/myBackupProviderView", model)
}
// we need to implement a renderer to use handlebars
Renderer<?> getRenderer() {
if(renderer == null) {
renderer = new HandlebarsRenderer("renderer", getPlugin().getClassLoader())
renderer.registerAssetHelper(getPlugin().getName())
renderer.registerNonceHelper(getMorpheus().getWebRequest())
renderer.registerI18nHelper(getPlugin(),getMorpheus())
}
return renderer
}
}
Morpheus Backup Provider
A full backup provider implementation may not be required in many cases. The Morpheus Backup Provider can be used to handle all the high level operations. The example below would allow Morpheus to manage the backup job and delegate the backup execution and restore to back to the plugin’s Backup Type Providers.
class MyBackupProvider extends MorpheusBackupProvider {
MyBackupProvider(Plugin plugin, MorpheusContext context) {
super(plugin, context)
MySnapshotBackupProvider mySnapshotBackupProvider = new MySnapshotBackupProvider(plugin, morpheus)
plugin.registerProvider(mySnapshotBackupProvider)
addScopedProvider(mySnapshotBackupProvider, "vmware", null)
}
}
See the DigitalOcean Plugin for a full example implementation of plugin that utilizes the Morpheus Backup Provider.
Catalog Layouts Plugin
It is becoming popular for some enterprises and managed service providers to expose simpler options when it comes to provisioning workloads. These could be used to target employees who are not highly technical or to further restrict what someone is allowed to order. Not only can they provision workloads like vms, and containers, but also execute operational tasks created by the administrator. Sometimes, it is necessary to further customize how a catalog detail page looks. There may be special ways of displaying information, or even the order form components need some advanced customization.
This plugin exposes the ability to control everything from the HTML used to render the catalog item, to the javascript that controls the form options. By default, we use a server-side handlebars template renderer however this can be completely customized if so desired.
Setup
Given the advanced nature of this plugin, it may be best to start with the sample plugin provided in the plugin sample repository. This plugin replicates the embedded layout functionality so acts as a great starting point. It even includes the javascript used for rendering the option types within it.
|
Tip
|
Reference the Sample Catalog Layout Plugin before making your own. |
The core of the plugin starts with the CatalogItemLayoutProvider which extends the common UIProvider. Most of the UI related plugin types have some commonalities. The primary difference is the command line arguments sent to the render() method.
/**
* Example TabProvider
*/
class StandardCatalogLayoutProvider extends AbstractCatalogItemLayoutProvider {
Plugin plugin
MorpheusContext morpheus
String code = 'catalog-item-standard'
String name = 'Standard Catalog Layout'
StandardCatalogLayoutProvider(Plugin plugin, MorpheusContext context) {
this.plugin = plugin
this.morpheus = context
}
/**
* Demonstrates building a TaskConfig to get details about the Server and renders the html from the specified template.
* @param server details of a ComputeServer
* @return
*/
@Override
HTMLResponse renderTemplate(CatalogItemType catalogItemType, User user) {
ViewModel<CatalogItemType> model = new ViewModel<>()
model.object = catalogItemType
getRenderer().renderTemplate("hbs/standardCatalogItem", model)
}
}
The render method allows the CatalogItemType model to be passed into a handlebars view for rendering.
The handlebars template, in this case, takes over the rendering of the entire page below the main navigation. It allows inclusion of external assets as well as assets included in the project via asset-pipeline.
<script src="{{asset "/form_manager.js"}}" ></script>
<link rel="stylesheet" type="text/css" href="{{asset "/styles.css"}}">
<script src="{{asset "/templates/plugin-configurable-option.js"}}" ></script>
<div class="page-content">
<div class="catalog-item-details">
<div class="item-type-header">
<div class="item-type-image">
<img class="item-header-img" src="{{catalogTypeImage}}" title="{{name}}" onError="loadImage(this);"/>
</div>
<h1 class="ellipsize" title="{{name}}">{{name}}</h1>
<div class="desc">{{description}}</div>
</div>
<div class="catalog-item-body break-container-sm">
{{#hasWiki}}
<div class="catalog-item-content wiki-content">
{{wiki}}
</div>
{{/hasWiki}}
<div class="catalog-item-configuration {{#hasWiki}}with-item-content{{/hasWiki}}">
{{#orderForm}}
<div class="actions text-right">
</div>
{{/orderForm}}
</div>
</div>
</div>
</div>
NOTE: A couple helper methods are registered such as the orderForm block which injects the order form data into the html render.
|
Tip
|
Pay special attention to the included javascript files used for rendering options. More often than not, one would want to copy these for use in a custom layout. |
Consuming the Plugin
Once a Catalog Layout plugin is compiled and loaded into a Morpheus environment, the layout is automatically made available globally for use when creating a service catalog item in the Blueprints section. Simply edit the catalog item and a new dropdown showing available layouts should be available to choose.
Cloud Provider Plugins
The Cloud provider plugin interfaces are among the more complicated plugin types to implement within Morpheus, but when implemented successfully it provides a very powerful method for creating custom clouds or even updating existing cloud functionality. There are two primary concepts that must first be discussed when developing a cloud plugin.
A Cloud plugin typically defines a cloud type (you may see this as zoneType in api for legacy compatibility.) as well as at least one ProvisionProvider. A Provision Provider (also seen in api as provisionType) defines how a resource is provisioned within a cloud. A cloud could offer many provisioning types. For example, Amazon offers both EC2 as well as RDS.
There are several other provider implementations that are needed on more advanced cloud implementations and some are not yet built out. This includes NetworkProvider and BackupProvider implementations as well as a few more.
Setup
Before Getting Started, It is recommended to look at the digital ocean sample plugin just to get some bearings. To get started you firstly will create a new class that implements com.morpheusdata.core.CloudProvider. The CloudProvider requires implementation of several methods including the code required to sync existing workloads from the cloud. All sync related code normally lives in here.
On a cloud implementation there are 2 scheduled jobs for refreshing. There is firstly a 5 minute sync job (typically) that syncs state changes on a periodic basis. Then secondly a daily job that runs at Midnight UTC to do larger syncs like price data or things that may change less frequently. Refer to implementations of the refresh() method as well as the refreshDaily() methods.
Finally one must also provide the option types inputs for configuring the add cloud wizard as well as the ComputeServerType objects available to this zone type. There should be multiple based often on platform and management state.
Defining Credentials
During the refresh() call (and others) the CloudProvider implementation will need to reach out to underlying cloud using credentials. There are two ways to define and store the required credentials.
Using OptionTypes
The simplest method is to define OptionType fields (like username and password) and return them on implementation of CloudProvider.getOptionTypes(). These will then be displayed on the Cloud configuration UI. The values can then be obtained during sync operations via the Cloud’s getConfigMap or directly on the object itself (i.e. serviceUsername) depending on how the OptionType was defined.
Using Morpheus Credentials
A more flexible option is to use Morpheus' built-in Credentials support. With this option, Credentials can be stored securely in Morpheus and utilized in various locations. In order to user this method, a few specific OptionType objects need to be defined. (Refer to VmwareCloudProvider for an example implementation)
-
An
OptionTypeneeds to be defined to represent the selection of the Credential type. The following properties must be configured on theOptionType:inputType=OptionType.InputType.CREDENTIAL,fieldContext=credential,fieldName=type,optionSource=credentials. In addition, theconfigfor theOptionTypeshould be something like'{"credentialTypes":["username-password"]}'. Where the array of types may be one or more ofusername-password, username-password-keypair, username-keypair, access-key-secret, client-id-secret, username-api-key, email-private-key, tenant-username-keypair, oauth2, api-key. These represent the preconfigured credential types in Morpheus. -
OptionTypeobjects need to be defined to represent the 'local' auth values. For example, username and password would need their ownOptionType. For these 'local' types, theirlocalCredentialvalue must betrue -
Any
OptionTypesthat should be reloaded when the Credential input changes should includecredential-typein theirdependsOnvalue. This will trigger theOptionTypeoptionSourcefunction to be called when the Credentials change.
To load the Credential information that may be set on a Cloud, the MorpheusCloudService can be used.
To load the Credential information from within OptionSourceService implementations (which may be called during Cloud configuration), MorpheusAccountCredentialService may be used to load Credential information from the passed in form options. See VmwareOptionSourceService for an example.
Datastore Type Providers
A Datastore Type Provider allows the plugin developer to implement custom Datastore types for use with various provision providers. (Currently only VME/MVM is supported). This could be for use with third party storage arrays or in the event some other type of Software Defined Storage (SDS) is desired to be used with the VME platform.
What is the difference between a Datastore Type Provider and a Storage Provider?
Storage Providers are typically used for managing storage outside the general provisioning workload pipeline. This allows for volume and share management within a datastore that can be tied into automation, but does not directly affect provisioning operations. Where as, a Datastore, is a storage target for a workload to run on. This provides additional methods that give context that can leverage the Storage Server (defined by the Storage Provider).
Getting Started
Before getting started, it is recommended that a StorageProvider also be created along with a DatastoreTypeProvider plugin.
This is because they often go together. For example, an Alletra Storage Array can be registered by the user as a Storage Integration
via the StorageProvider and then the DatastoreTypeProvider can be used to define the various types of Datastore objects that can be created
using this storage array (i.e. iSCSI LUN per vDISK).
Another common provider type that goes along with this is the BackupProvider. Any type of custom backup implementation
should leverage that provider along with this.
NOTE: Pay special attention to the Facet options on the interface. These provide additional functionality when they
are implemented on the provider class.
Create a new class that implements com.morpheusdata.core.providers.DatastoreTypeProvider and implement the required methods as well as the `StorageProvider.
class MyCustomDatastoreTypeProvider implements DatastoreTypeProvider, DatastoreTypeProvider.MvmProvisionFacet, DatastoreTypeProvider.SnapshotFacet.SnapshotServerFacet {
@Override
String getStorageProviderCode() {
return 'my-storage-provider-code'
}
// Implement required methods here for all 3 interfaces
}
class MyCustomStorageProvider implements StorageProvider {
@Override
String getCode() {
return 'my-storage-provider-code'
}
}
Your favorite IDE (such as IntelliJ IDEA) should be able to auto-generate the required methods for you.
Or, you can use the Getting Started plugin generator on the developer portal to create a new Datastore Type Provider.
Facets
In the above example, two additional facets were injected into the class. These control implementation of Snapshots as well as the ability to inject behaviors specific to VME/MVM Provisioning. For example, in some scenarios it may be necessary to prepare the host for a vm move or even an initial provision of a vm. These allow for injection points for this.
The MVMProvisionFacet also contains a means to override/customize the libvirt Domain XML block device specification. This
can be important if using a custom storage array implementation that may use raw BLOCK or iSCSI vs QCOW2 formats.
Host Interactions
When utilizing the DatastoreTypeProvider for VME/MVM it is often necessary to execute automation directly on the hypervisor.
It is recommended that this be performed with the morpheusContext.executeCommandOnServer method. This will automatically ensure
the command is executed through the right medium (such as through an Edge Distributed Worker and through the agent with proper ssh fallback).
The host of a specific vm/workload is normally found by grabbing the ComputeServer.getParentServer() method off the workload server representation.
Snapshots
Snapshots are a common feature of most storage arrays and SDS solutions. The SnapshotFacet provides a means to implement
as well as the SnapshotServerFacet which is more commonly used. The key difference is one implementation focuses strictly on per volume snapshots, and the other focuses on snapshotting all volumes within a server at once. This is useful for things like backup operations, image export, or just general state.
Generic Integration Plugins
Generic Integration Plugins enable the registration of a "Generic" AccountIntegrationType. A Generic Integration plugin is designed to add additional functionality to existing providers and extend the capability of Morpheus where existing providers are not available.
For example, a Jenkins plugin implementing a Generic Integration Provider could store credentials for Jenkins task types, allowing users to avoid entering credentials repeatedly when creating new tasks. Additionally, the plugin could periodically sync data, such as a list of projects, used the DatasetProvider dropdowns in the Jenkins task types.
A Generic Integration example can be found in the sample plugin repository on GitHub.
Network Provider Plugins
The Network Provider Plugin interface requires either an associated GenericIntegrationProvider or CloudProvider provider in the plugin to provide the account integration or cloud backing for the network server as well as sharing icons and other resources. Make sure to set the appropriate code in the NetworkProvider implementation.
IPAM/DNS Plugins
The IPAM and DNS Provider Plugin interfaces provide an easy means to create direct orchestration for IP Address allocation/release as well as DNS Name server registrations. An IPAM Provider is often capable of implementing both interfaces as they often provide both services at once. It is also possible to independently register a DNSProvider only.
Both the DNS and IPAM Provider plugins typically consist of just 3 parts. First is defining the provider information such as the configuration options for adding the integration as well as pool types it offers. Secondly periodically syncing state data into morpheus back from the remote integration endpoints. This could include just syncing in available pools or zones, all the way to syncing in all IP host records or DNS zone records. This is up to your implementation.
Setup
Before Getting Started, It is recommended to look at the infoblox plugin just to get some bearings. To get started you firstly will create a new class that implements com.morpheusdata.core.IPAMProvider and com.morpheusdata.core.DNSProvider. The Providers requires implementation of several methods including the code required to sync existing records. All sync related code normally lives in here.
Both providers also require implementing CRUD based methods for creating host records, deleting host records, and creating zone records and deleting zone records. It is important to note that the host record object allows a user to directly enter an ip address to be requested for allocation or, if none is provided, it should be assumed the next available IP should be acquired. Host records are also special in that there are additional options for simultaneously creating DNS records such as A and PTR records. The additional complexity of tieing these pieces into the automated provisioning of workloads is hidden and taken care of by the Morpheus orchestrator.
Reports Plugin
Create custom report types for users to consume within Morpheus. Customize the behavior of how the report data is assembled and generated as well as the way it is rendered/displayed to the user. See a full example.
Setup
Creating a report is just a matter of registering a new implementation of a ReportProvider. If using standard handlebars rendering similar to UI Extensions, simply extend the com.morpheusdata.core.AbstractReportProvider. Before creating a report a few concepts should be made known.
There are a few model objects of importance. Firstly, the ReportProvider implementation always generates a ReportType for reference and display when the user is browsing a report ro run.
After a report is run a ReportResult is generated. This represents the information of who created the report as well as any submitted filters / report options related to the report. When creating a process method the results of the run should be stored as ReportResultRow objects. These have a displayOrder and section. This allows one to store header data as well as line item data for rendering and csv export. These rows are generated using rxjava asynchronous flows in the process method. Example Here:
void process(ReportResult reportResult) {
morpheus.report.updateReportResultStatus(reportResult,ReportResult.Status.generating).blockingGet()
Long displayOrder = 0
List<GroovyRowResult> results = []
withDbConnection { Connection dbConnection ->
if(reportResult.configMap?.phrase) {
String phraseMatch = "${reportResult.configMap?.phrase}%"
results = new Sql(dbConnection).rows("SELECT id,name,status from instance WHERE name LIKE ${phraseMatch} order by name asc;")
} else {
results = new Sql(dbConnection).rows("SELECT id,name,status from instance order by name asc;")
}
}
Observable<GroovyRowResult> observable = Observable.fromIterable(results) as Observable<GroovyRowResult>
observable.map{ resultRow ->
Map<String,Object> data = [name: resultRow.name, id: resultRow.id, status: resultRow.status]
ReportResultRow resultRowRecord = new ReportResultRow(section: ReportResultRow.SECTION_MAIN, displayOrder: displayOrder++, dataMap: data)
return resultRowRecord
}.buffer(50).doOnComplete {
morpheus.report.updateReportResultStatus(reportResult,ReportResult.Status.ready).blockingGet()
}.doOnError { Throwable t ->
morpheus.report.updateReportResultStatus(reportResult,ReportResult.Status.failed).blockingGet()
}.subscribe {resultRows ->
morpheus.report.appendResultRows(reportResult,resultRows).blockingGet()
}
}
NOTE: Notice that this process method features the ability to get a read only database connection to the morpheus MySQL Database. This isn’t always the best option but is a good fallback option for grabbing data you may not otherwise be able to get. Other data query methods are available on the various MorpheusContext subService classes. Expect more of these to be filled out as the plugin ecosystem develops. A good example of this is the MorpheusAccountInvoiceService found via the MorpheusCostingService. It enables you to query all invoices just as you would from the api.
Custom Filters
It is often the case that a user may want to adjust how a filter runs. Perhaps they want to reduce the result set to a filtered set of data or group by certain properties. For this, the ReportProvider provides a getOptionTypes method that when implemented allows the developer to specify custom form inputs the user has to select when running the report.
@Override
List<OptionType> getOptionTypes() {
[new OptionType(code: 'status-report-search', name: 'Search', fieldName: 'phrase', fieldContext: 'config', fieldLabel: 'Search Phrase', displayOrder: 0)]
}
It is important to note the fieldContext should almost always be set to config in this instance.
Rendering
Render is very similar to rendering a tab. The main difference is the payload that is sent for the render is the ReportResult representing the particular report run as well as the dataset rows grouped by section.
@Override
HTMLResponse renderTemplate(ReportResult reportResult, Map<String, List<ReportResultRow>> reportRowsBySection) {
ViewModel<String> model = new ViewModel<String>()
model.object = reportRowsBySection
getRenderer().renderTemplate("hbs/instanceReport", model)
}
|
Tip
|
When using custom javascript or stylesheets be sure to use the provided {{nonce}} helper to inject the appropriate nonce token for the Content-Security-Policy.
|
Storage Datastore Plugin for HVM
Building a storage plugin for the Morpheus HVM (KVM-based) hypervisor involves two primary provider types working together: a StorageProvider that manages the external storage integration (e.g., a SAN or SDS array), and a DatastoreTypeProvider that integrates that storage into the HVM provisioning lifecycle. This guide walks through the design and implementation of a block-based storage datastore plugin targeting Software Defined Storage (SDS) solutions such as StorPool, HPE Alletra, or similar platforms.
Architecture Overview
A storage plugin for HVM typically consists of the following components registered within a single Plugin class:
-
StorageProvider — Manages the external storage integration (the "Storage Server"). Handles connectivity, authentication, periodic sync/refresh, volume CRUD for the storage management UI, and custom option types.
-
DatastoreTypeProvider — Defines how the storage server participates in the HVM provisioning lifecycle. Handles datastore creation on clusters, volume creation/clone/resize/remove during VM provisioning, host preparation (LUN mapping, device discovery), and libvirt disk configuration.
-
DatasetProviders — Supply dynamic dropdown data for option types in the UI (e.g., lists of storage pools, protocol types, available arrays).
-
BackupProvider (optional) — Provides snapshot-based backup and restore using the storage array’s native capabilities.
The relationship between these providers is important. The DatastoreTypeProvider references the StorageProvider via its getStorageProviderCode() method. This allows a user to first register the storage array as a Storage Integration (via the StorageProvider), then create Datastores on HVM clusters that use that array (via the DatastoreTypeProvider).
Directory-Based vs Block-Based Storage
HVM supports multiple storage paradigms. Understanding the distinction between directory-based and block-based storage is critical when designing your plugin.
Directory-Based (File) Storage
Directory-based datastores (e.g., local directory, NFS, CIFS, GFS2, OCFS2) store virtual machine disks as files — typically in QCOW2 format — within a directory path on the hypervisor host. The libvirt XML references these as <disk type="file"> or <disk type="volume"> entries.
Key characteristics:
-
Disk format is usually
qcow2, supporting thin provisioning and snapshot chains (backing files) -
Files live under a
datastore.externalPathdirectory (e.g.,/var/morpheus/kvm/images) -
QCOW2 images can be cloned via
qemu-img convertorcp -
Snapshots can leverage QCOW2 backing chains with
virsh snapshot-create-as --disk-only -
Each VM gets a subdirectory under the datastore path:
{externalPath}/{serverExternalId}/{volumeName}
Block-Based Storage
Block-based datastores present raw block devices (LUNs/volumes) directly to hypervisor hosts. Each virtual disk maps to a dedicated block device on the storage array. The libvirt XML references these as <disk type="block"> entries.
Key characteristics:
-
Disk format is
raw(no QCOW2 overhead) -
Each volume maps to a device on the hypervisor (e.g.,
/dev/dm-Xfor multipath or/dev/disk/by-id/…) -
Cloning is typically performed by the storage array itself (array-side clone) or by creating a new volume and copying data via
qemu-img convert -
Snapshots are typically array-native (e.g., thin clones, crash-consistent snapshots)
-
Requires host preparation after volume creation: LUN mapping/export, SCSI rescan, multipath device discovery
NOTE: Block devices (vDISKs) must be accessible from ALL hypervisor hosts in the cluster at all times. The HVM high-availability failover code does not invoke any plugin hooks to restore, transfer, or re-map a LUN to a different host during a failover event. When a hypervisor fails and its VMs are restarted on a surviving host, the HA controller simply redefines and starts the VM on the new host using the existing libvirt XML. If the block device referenced in that XML is not already visible on the new host, the VM will fail to start. For SAN/block-based storage this means all LUNs must be exported/mapped to every hypervisor in the cluster (not just the host currently running the VM). For network-attached block storage (e.g., Ceph RBD), this is inherently satisfied because all hosts connect to the same Ceph cluster. For FC/iSCSI storage, your plugin’s createDatastore() and ADD_WORKER event handler must ensure that every host’s initiators are registered with the storage array and that all volumes are accessible from every host.
Why You Need Both
A block-based storage plugin should also provide a companion directory-based datastore (or use an existing one) for storing shared image files. Virtual images in Morpheus are typically cached as QCOW2 files before being cloned to their target storage. When using block storage, the provisioning flow needs a place to store and read these source images for the initial qemu-img convert from QCOW2 into the raw block device.
For example, if you are implementing a block storage plugin for a solution like StorPool:
-
The directory-based datastore (NFS, GFS2, or a shared filesystem) stores the QCOW2 image cache shared across the cluster
-
The block-based datastore (your plugin) creates raw volumes on the SDS array and maps them to hosts for VM provisioning
-
During clone operations, your plugin reads the source QCOW2 from the directory datastore and writes it to the new block device using
qemu-img convert
If your SDS solution also supports a POSIX filesystem mode (as many SDS platforms do), you can implement a single plugin that provides both a directory-based datastore type and a block-based datastore type, each backed by the same storage cluster.
Project Setup
A storage plugin project follows the standard Morpheus plugin structure. Your build.gradle should include the morpheus-plugin-api dependency and any client libraries needed to communicate with your storage array.
plugins {
id "com.github.johnrengelman.shadow" version "8.1.1"
}
apply plugin: 'java'
apply plugin: 'groovy'
group = 'com.example.storage'
version = '1.0.0'
sourceCompatibility = '11'
targetCompatibility = '11'
dependencies {
compileOnly 'com.morpheusdata:morpheus-plugin-api:1.3.0'
compileOnly 'org.codehaus.groovy:groovy-all:3.0.21'
// Your SDS client library
implementation 'com.example:storpool-client:1.0.0'
}
jar {
manifest {
attributes(
'Plugin-Class': 'com.example.storage.MyStoragePlugin',
'Plugin-Version': archiveVersion.get(),
'Morpheus-Name': 'My SDS Storage',
'Morpheus-Organization': 'My Organization',
'Morpheus-Code': 'my-sds-storage-plugin',
'Morpheus-Description': 'SDS Storage Integration for HVM',
'Morpheus-Min-Appliance-Version': '8.0.0'
)
}
}
tasks.assemble.dependsOn tasks.shadowJar
Plugin Class
The plugin class registers all providers. The order of registration does not matter, but the relationship between the StorageProvider code and DatastoreTypeProvider.getStorageProviderCode() must match.
class MyStoragePlugin extends Plugin {
@Override
String getCode() {
return 'my-sds-storage-plugin'
}
@Override
void initialize() {
this.name = 'My SDS Storage'
this.description = 'SDS Storage Integration for HVM'
// Register the storage server integration provider
this.registerProvider(new MySdsStorageProvider(this, this.morpheus))
// Register the HVM datastore type provider
this.registerProvider(new MySdsDatastoreProvider(this, this.morpheus))
// Register any dataset providers for dynamic UI dropdowns
this.registerProvider(new StoragePoolDatasetProvider(this, this.morpheus))
}
@Override
void onDestroy() {
// Cleanup resources
}
}
Implementing the StorageProvider
The StorageProvider manages the storage server integration lifecycle. It defines how the storage array appears under Infrastructure > Storage and handles connectivity, authentication, and periodic data sync.
class MySdsStorageProvider extends AbstractStorageProvider
implements StorageProviderVolumes {
MorpheusContext morpheusContext
Plugin plugin
MySdsStorageProvider(Plugin plugin, MorpheusContext morpheusContext) {
super(plugin, morpheusContext)
this.morpheusContext = morpheusContext
this.plugin = plugin
}
@Override
String getCode() {
return 'my-sds-storage'
}
@Override
String getName() {
return 'My SDS Storage'
}
@Override
String getDescription() {
return 'SDS Storage Array Integration'
}
@Override
Icon getIcon() {
return new Icon(path: 'my-sds-logo.svg', darkPath: 'my-sds-logo-dark.svg')
}
@Override
StorageServerType getStorageServerType() {
return new StorageServerType(
code: 'my-sds-storage',
name: 'My SDS Storage',
description: 'SDS Storage Array',
hasBlock: true,
hasObject: false,
hasFile: false,
hasDisk: true,
optionTypes: getStorageServerOptionTypes(),
volumeTypes: getStorageVolumeTypes()
)
}
@Override
ServiceResponse verifyStorageServer(StorageServer storageServer, Map opts) {
// Validate connectivity and credentials against the SDS API
// Return ServiceResponse.error() with field-level errors if validation fails
}
@Override
ServiceResponse initializeStorageServer(StorageServer storageServer, Map opts) {
// Called on first save. Perform initial sync of pools, volumes, etc.
return refreshStorageServer(storageServer, opts)
}
@Override
ServiceResponse refreshStorageServer(StorageServer storageServer, Map opts) {
// Periodic sync: update capacity, discover volumes, sync pools
// Use SyncTask for efficient delta sync of large datasets
}
// StorageProviderVolumes methods for managing volumes from the Storage UI
@Override
Collection<StorageVolumeType> getStorageVolumeTypes() {
return [
new StorageVolumeType(
code: 'my-sds-block-volume',
name: 'SDS Block Volume',
displayOrder: 0
)
]
}
// ... implement createVolume, deleteVolume, resizeVolume, updateVolume ...
}
NOTE: The StorageServerType.code must match what DatastoreTypeProvider.getStorageProviderCode() returns. This is how Morpheus links a datastore to its backing storage integration.
Implementing the DatastoreTypeProvider
The DatastoreTypeProvider is where the real provisioning integration happens. This provider defines how your storage participates in the HVM VM lifecycle — from datastore creation on a cluster, through volume provisioning, to teardown.
For block-based SDS, you will typically implement the following interfaces:
-
DatastoreTypeProvider— Core datastore lifecycle (create, update, remove datastores; create, clone, resize, remove volumes) -
DatastoreTypeProvider.MvmProvisionFacet— HVM-specific hooks for host preparation and libvirt disk configuration -
DatastoreTypeProvider.SnapshotFacet.SnapshotServerFacet— Server-level snapshot operations (recommended for HVM) -
PluginProvider.EventSubscriberFacet— React to cluster events (worker add/remove, volume attach/detach)
class MySdsDatastoreProvider implements
DatastoreTypeProvider,
DatastoreTypeProvider.MvmProvisionFacet,
DatastoreTypeProvider.SnapshotFacet.SnapshotServerFacet,
PluginProvider.EventSubscriberFacet {
MorpheusContext morpheusContext
Plugin plugin
MySdsDatastoreProvider(Plugin plugin, MorpheusContext morpheusContext) {
this.morpheusContext = morpheusContext
this.plugin = plugin
}
@Override
MorpheusContext getMorpheus() { return morpheusContext }
@Override
Plugin getPlugin() { return plugin }
@Override
String getCode() { return 'my-sds-hvm-block' }
@Override
String getName() { return 'My SDS Block Storage HVM' }
@Override
String getProvisionTypeCode() { return 'kvm' }
@Override
String getStorageProviderCode() { return 'my-sds-storage' }
@Override
boolean getCreatable() { return true }
@Override
boolean getEditable() { return true }
@Override
boolean getRemovable() { return true }
@Override
List<OptionType> getOptionTypes() {
// Return option types for the datastore creation dialog
// e.g., storage server selector, pool selector, protocol type
}
@Override
List<StorageVolumeType> getVolumeTypes() {
// Return the volume types available for this datastore
}
// ... lifecycle methods follow ...
}
Datastore Lifecycle
Creating a Datastore
When a user creates a new datastore on an HVM cluster, Morpheus calls validateDatastore() followed by createDatastore(). For block storage, the create operation typically involves:
-
Validating the selected storage server and configuration options
-
Iterating over all hypervisor hosts in the cluster
-
Registering each host with the storage array (e.g., creating a host set or initiator group with the hosts' iSCSI IQNs or FC WWPNs)
-
Tracking per-host status using
DatastoreLocationentries
@Override
ServiceResponse validateDatastore(Datastore datastore) {
ServiceResponse rtn = ServiceResponse.success()
if (!datastore.name) {
return ServiceResponse.error('Datastore name is required')
}
if (!datastore.storageServer) {
rtn.errors['storageServerId'] = 'Storage server is required'
}
// Validate protocol type, pool selection, etc.
if (rtn.errors) {
rtn.success = false
}
return rtn
}
@Override
ServiceResponse<Datastore> createDatastore(Datastore datastore) {
// Fetch the cluster and its hosts
ComputeServerGroup cluster = morpheusContext.services.cluster.get(
datastore.refId
)
List<ComputeServer> hosts = morpheusContext.services.computeServer.list(
new DataQuery().withFilter('serverGroup.id', datastore.refId)
)
// For each host, gather path information (IQNs, WWPNs)
// and register them with the storage array
// (host set / initiator group)
// Update the datastore with storage-side metadata
// (e.g., host set name, array serial number)
return ServiceResponse.success(datastore)
}
NOTE: The Datastore.refId for HVM datastores points to the cluster (ComputeServerGroup) ID. Use morpheusContext.services.cluster.get(datastore.refId) to retrieve the cluster.
Refreshing a Datastore
The refreshDatastore() method is called periodically to sync the datastore state. Use this to update capacity metrics, verify host registrations, and clean up stale resources.
Removing a Datastore
When removing a datastore, verify no volumes are still attached before allowing removal. Clean up any host registrations on the storage array.
@Override
ServiceResponse removeDatastore(Datastore datastore) {
// Check for attached volumes
Long volumeCount = morpheusContext.services.storage.volume.count(
new DataQuery().withFilter('datastore.id', datastore.id)
)
if (volumeCount > 0) {
return ServiceResponse.error(
'Cannot remove datastore: volumes are still attached'
)
}
// Clean up host set / initiator group on storage array
return ServiceResponse.success()
}
Volume Lifecycle During Provisioning
The volume lifecycle is the heart of the integration. When Morpheus provisions a VM on an HVM cluster, it calls your provider’s volume methods in a specific order.
Provisioning Flow (Create/Clone VM)
-
createVolume()orcloneVolume()— Called for each disk on the VM. For the root disk,cloneVolume()is called with a source volume (the cached QCOW2 image). For additional data disks,createVolume()is called. -
prepareHostForVolume()(MvmProvisionFacet) — After all volumes are created, this is called for each volume to prepare the hypervisor host. For block storage, this is where you map/export the LUN to the host and discover the block device. -
buildDiskConfig()(MvmProvisionFacet) — Called to generate the libvirt XML disk specification. Your provider returns the disk type (block), device type, and disk mode. -
VM is defined and started via libvirt.
Teardown Flow (Destroy VM)
-
VM is stopped via libvirt.
-
removeVolume()— Called for each volume. Unexport/unmap the LUN from the host and delete the volume on the storage array. -
releaseVolumeFromHost()(MvmProvisionFacet) — Called after volume removal to perform any host-side cleanup (e.g., multipath device removal, SCSI rescan).
Creating a Volume
For block storage, createVolume() provisions a new LUN/volume on the storage array.
@Override
ServiceResponse<StorageVolume> createVolume(
StorageVolume storageVolume, ComputeServer computeServer) {
// 1. Determine volume size, name, and storage pool
// from the datastore config
// 2. Call the SDS API to create a new volume/LUN
// 3. Store the volume's external identifier (WWN, UID, etc.)
// on the StorageVolume record for later use
// 4. Save the volume
return ServiceResponse.success(storageVolume)
}
Cloning a Volume
Clone operations are central to HVM provisioning. When a VM is provisioned from a virtual image, the source QCOW2 file must be written to the new block device. There are two common approaches:
Approach 1: qemu-img convert (QCOW2 to block device)
This is the most common approach when the source image is a QCOW2 file stored on a directory-based datastore. Create the destination volume on the array, map it to the hypervisor, then use qemu-img convert to copy the data.
@Override
ServiceResponse<StorageVolume> cloneVolume(
StorageVolume volume, ComputeServer server,
StorageVolume sourceVolume) {
// 1. Create a new volume on the storage array
// 2. Map/export the volume to the hypervisor host
// 3. Discover the multipath/block device on the host
// 4. Run qemu-img convert on the hypervisor:
// qemu-img convert -f qcow2 -O raw \
// {source_qcow2_path} {block_device}
// Use morpheusContext.executeCommandOnServer(host, command)
// 5. Update the StorageVolume with the block device path
return ServiceResponse.success(volume)
}
Approach 2: Array-side clone (block to block)
If the source volume is already a block device on the same storage array (e.g., a previously cached image or another block volume), you can use the array’s native clone/snapshot capability for a faster copy.
@Override
ServiceResponse<StorageVolume> cloneVolume(
StorageVolume volume, ComputeServer server,
StorageVolume sourceVolume) {
// Check if source is on the same array (has a WWN/UID)
if (isBlockVolumeOnSameArray(sourceVolume)) {
// Use array-side clone API for fast copy
// Update volume metadata with the clone's identifiers
} else {
// Fall back to qemu-img convert approach
}
return ServiceResponse.success(volume)
}
NOTE: The cloneVolume(StorageVolume volume, ComputeServer server, VirtualImage virtualImage, CloudFileInterface cloudFile) overload is called when there is no local image cache and the image must be streamed directly. For block storage, you would typically create the volume, then stream the image data using the MorpheusFileCopyService.
Resizing a Volume
Volume resize for block storage involves:
-
Calling the storage array API to expand the LUN
-
Rescanning/resizing the multipath device on all cluster hosts
-
Notifying the guest VM of the new size via
virsh blockresize
@Override
ServiceResponse<StorageVolume> resizeVolume(
StorageVolume storageVolume,
ComputeServer computeServer, Long newSize) {
// 1. Validate: block storage typically cannot shrink
if (storageVolume.maxStorage > newSize) {
return ServiceResponse.error(
'Volume shrinking is not supported'
)
}
// 2. Resize on the storage array
// 3. Rescan multipath devices on cluster hosts
// 4. virsh blockresize to notify the guest VM
// 5. Update storageVolume.maxStorage
return ServiceResponse.success(storageVolume)
}
Removing a Volume
Volume removal should unmap the LUN from all hosts, remove multipath devices, and delete the volume on the array.
@Override
ServiceResponse removeVolume(
StorageVolume storageVolume,
ComputeServer computeServer,
boolean removeSnapshots, boolean force) {
// 1. Unmap/unexport the volume from the host(s)
// 2. Remove multipath devices on the hypervisor
// 3. Delete the volume on the storage array
// (optionally remove snapshots based on the flag)
return ServiceResponse.success()
}
MvmProvisionFacet: Host Preparation and Disk Configuration
The MvmProvisionFacet provides critical hooks for block-based storage. Without these, the HVM provisioner would not know how to present your block devices to VMs.
prepareHostForVolume
This method is called after volume creation and before VM definition. For block storage, this is where you:
-
Map/export the LUN to the hypervisor host (if not already done during createVolume)
-
Trigger a SCSI rescan or iSCSI session refresh on the host
-
Discover the multipath device path
-
Set the
StorageVolume.externalIdto the device path for use in libvirt XML
@Override
ServiceResponse<StorageVolume> prepareHostForVolume(
ComputeServerGroup cluster, ComputeServer server,
StorageVolume volume) {
// The 'server' here is the workload/VM, not the hypervisor.
// Get the hypervisor via server.parentServer
ComputeServer hypervisor =
morpheusContext.services.computeServer.get(
server.parentServer.id
)
// 1. Rescan for new LUNs on the hypervisor
// e.g., echo "1" > /sys/class/fc_host/hostX/issue_lip
// or iscsiadm session rescan
morpheusContext.executeCommandOnServer(
hypervisor, rescanCommand
)
// 2. Discover the multipath device by WWN or volume UID
// e.g., /dev/disk/by-id/dm-uuid-mpath-XXXX
// or /dev/mapper/mpathX
String devicePath = discoverMultipathDevice(
hypervisor, volume
)
// 3. Set the external ID and device name on the volume
volume.externalId = devicePath
volume.setConfigProperty('source.dev', devicePath)
return ServiceResponse.success(volume)
}
NOTE: For multi-host clusters, you should prepare the volume on all hypervisors in the cluster (not just the target host), since VMs may live-migrate between hosts. Run the host preparation in parallel across all cluster hosts, but ensure the provisioning host’s result is returned.
buildDiskConfig
This method tells the HVM provisioner how to represent the volume in the libvirt domain XML. For block storage:
@Override
ServiceResponse<MvmDiskConfig> buildDiskConfig(
ComputeServerGroup cluster, ComputeServer server,
StorageVolume volume) {
MvmDiskConfig config = new MvmDiskConfig()
if (volume.type?.category == 'cd') {
config.deviceType = MvmDiskConfig.DeviceType.CDROM
config.diskMode = MvmDiskConfig.DiskMode.SATA
} else {
config.deviceType = MvmDiskConfig.DeviceType.DISK
config.diskMode = MvmDiskConfig.DiskMode.VIRTIO
}
config.diskType = 'block'
return ServiceResponse.success(config)
}
When diskType is set to block, the HVM provisioner generates a libvirt XML disk element like:
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none"
io="io_uring" discard="unmap"/>
<source dev="/dev/dm-uuid-mpath-XXXX"/>
<target dev="vda" bus="virtio"/>
</disk>
The source dev value comes from the volume’s config map (configMap.source.dev) or falls back to externalId. You can control this by setting properties on the volume during prepareHostForVolume().
For directory-based datastores, the libvirt XML uses <disk type="file"> instead and the source element references a file path:
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"
io="io_uring" discard="unmap"/>
<source file="/mnt/shared-storage/vm-123/disk-0.qcow2"/>
<target dev="vda" bus="virtio"/>
</disk>
This distinction in libvirt XML is entirely driven by what your buildDiskConfig() returns for diskType. The value block triggers <disk type="block"> with raw format, while file triggers <disk type="file"> with qcow2 format.
releaseVolumeFromHost
Called when a VM is being moved off a host or during teardown. For block storage, you may choose to defer cleanup to removeVolume() instead, or perform host-side device cleanup here.
@Override
ServiceResponse<StorageVolume> releaseVolumeFromHost(
ComputeServerGroup cluster, ComputeServer server,
StorageVolume volume) {
// For block storage, actual volume deletion is handled
// by removeVolume(). This hook is for host-side cleanup
// only (optional).
return ServiceResponse.success(volume)
}
Event Handling
Implementing PluginProvider.EventSubscriberFacet (or DatastoreTypeProvider.DatastoreEventFacet) allows your plugin to react to cluster and datastore events. This is important for maintaining host registrations when cluster membership changes.
@Override
List<EventType> getSupportedEventTypes() {
return [
EventType.VOLUME_ATTACH,
EventType.VOLUME_DETACH,
EventType.ADD_WORKER,
EventType.REMOVE_WORKER
]
}
@Override
ServiceResponse onEvent(Event event) {
switch (event) {
case ClusterEvent:
// ADD_WORKER: Register new host with storage array
// (add initiators to host set)
// REMOVE_WORKER: Unregister host from storage array
break
case DatastoreEvent:
// VOLUME_ATTACH: Ensure volume is mapped to host
// VOLUME_DETACH: Clean up volume mapping from host
break
}
return ServiceResponse.success()
}
Worker add/remove events are particularly important for block storage. When a new hypervisor joins the cluster, you need to register its initiators (IQNs/WWPNs) with the storage array so that existing LUNs become accessible. When a hypervisor is removed, you should clean up its registration.
Executing Commands on Hypervisors
Throughout the plugin, you will frequently need to execute commands on hypervisor hosts for operations such as SCSI rescans, multipath device discovery, qemu-img operations, and virsh commands. Always use the morpheusContext.executeCommandOnServer() method rather than direct SSH.
// Execute a command on the hypervisor host
ComputeServer hypervisor = server.parentServer
TaskResult result = morpheusContext.executeCommandOnServer(
hypervisor,
'multipathd show paths format "%w %d %s"'
).blockingGet()
if (result.success) {
String output = result.data
// Parse multipath output
}
This method automatically routes through the correct communication channel — whether that is the morpheus agent, an edge gateway worker, or direct SSH — and handles authentication transparently.
Snapshots
Storage arrays with native snapshot capabilities can provide significantly faster and more efficient snapshot operations than the default QCOW2-based snapshots used by directory datastores. This section covers how to implement snapshot support in your storage datastore plugin.
Snapshot Facets
The DatastoreTypeProvider defines three snapshot facets. Choose the one that matches your use case:
-
SnapshotFacet— Volume-level snapshots. Operates on individualStorageVolumerecords. This is the lowest-level facet and is not commonly used directly for HVM. -
SnapshotFacet.SnapshotServerFacet— Server-level snapshots. Operates on aComputeServerand snapshots all of its volumes together. This is the recommended facet for HVM/MVM block storage because snapshots typically need to be crash-consistent across all volumes on a VM. -
SnapshotFacet.SnapshotInstanceFacet— Instance-level snapshots. Operates on anInstanceand its associatedCreateSnapshotRequest. Use this if your storage array supports application-consistent snapshots at the instance level, or if you need to coordinate snapshots across multiple servers in an instance.
For a block-based SDS plugin targeting HVM, implement SnapshotServerFacet on your DatastoreTypeProvider class. If you do not intend to support instance-level snapshots, you can also implement SnapshotInstanceFacet with stub methods that return an error — this prevents fallback to default behavior and gives users a clear message:
class MySdsDatastoreProvider implements
DatastoreTypeProvider,
DatastoreTypeProvider.MvmProvisionFacet,
DatastoreTypeProvider.SnapshotFacet.SnapshotServerFacet,
DatastoreTypeProvider.SnapshotFacet.SnapshotInstanceFacet,
PluginProvider.EventSubscriberFacet {
// ... other methods ...
// SnapshotInstanceFacet stubs (not supported)
@Override
ServiceResponse<Snapshot> createSnapshot(
Instance instance, CreateSnapshotRequest request) {
return ServiceResponse.error(
'Instance-level snapshots are not supported'
)
}
@Override
ServiceResponse<Snapshot> revertSnapshot(
Instance instance, Snapshot snapshot) {
return ServiceResponse.error(
'Instance-level snapshots are not supported'
)
}
@Override
ServiceResponse removeSnapshot(
Instance instance, Snapshot snapshot) {
return ServiceResponse.error(
'Instance-level snapshots are not supported'
)
}
}
Division of Responsibility: UI vs Plugin
Understanding what the Morpheus UI creates versus what your plugin creates is critical for correct snapshot implementation.
When a snapshot is requested:
-
The UI creates a parent
Snapshotdomain record in the database (with statuscreating, linked to the server and account). -
The UI calls your plugin’s
createSnapshot()method. -
Your plugin creates a
Snapshotmodel object and populates it withSnapshotFileentries — one per volume that was snapshotted. These are model objects, not yet persisted. -
Your plugin returns the
Snapshotin theServiceResponse. -
The UI takes the returned
SnapshotFileentries from your response, enriches them with additional metadata, and attaches them to the parent domain record it created in step 1. It then marks the snapshot ascompleteandcurrentlyActive.
The key takeaway: your plugin does not need to persist snapshots or snapshot files to the database. Build the model objects, populate them with the right data, and return them. The UI handles persistence.
If a VM has volumes spanning multiple datastore types (e.g., some on your block datastore and some on a directory datastore), the UI will call each datastore type’s snapshot provider separately and aggregate the results into a single parent snapshot. Your plugin only needs to handle volumes that belong to your datastore type.
Creating a Snapshot
The createSnapshot() method receives the server (VM), and two flags indicating the purpose of the snapshot. Your implementation should:
-
Iterate the server’s volumes and filter to only those on your datastore type
-
Create snapshot(s) on the storage array for those volumes
-
Build a
Snapshotmodel withSnapshotFileentries for each snapshotted volume -
If
forExportis true, export the snapshot volumes to the hypervisor so the backup agent can read the data
@Override
ServiceResponse<Snapshot> createSnapshot(
ComputeServer server,
Boolean forBackup, Boolean forExport) {
// 1. Get the server's volumes on our datastore
List<StorageVolume> volumes = server.volumes?.findAll { vol ->
vol.type?.category != 'cd' &&
vol.datastore?.type?.code == getCode()
}
if (!volumes) {
return ServiceResponse.error(
'No eligible volumes found for snapshot'
)
}
// 2. Create snapshot(s) on the storage array
// For arrays that support volume-set snapshots,
// snapshot all volumes together for crash consistency
def arraySnapshotResult = sdsClient.createSnapshotSet(
volumes.collect { getArrayVolumeId(it) }
)
if (!arraySnapshotResult.success) {
return ServiceResponse.error(
"Failed to create snapshot: ${arraySnapshotResult.error}"
)
}
// 3. Build the Morpheus Snapshot model
Snapshot snapshot = new Snapshot(
name: arraySnapshotResult.snapshotSetName,
externalId: arraySnapshotResult.snapshotSetUid
)
volumes.eachWithIndex { vol, idx ->
def snapVolInfo = arraySnapshotResult
.snapshotVolumes[idx]
SnapshotFile snapshotFile = new SnapshotFile(
name: snapVolInfo.name,
externalId: snapVolInfo.uid,
type: 'block',
volume: vol,
snapshot: snapshot,
diskIndex: vol.displayOrder ?: idx
)
// 4. If forExport, map the snapshot volume to the
// hypervisor and set exportPath so the morphd backup
// agent can read the snapshot data
if (forExport) {
ComputeServer hypervisor =
morpheusContext.services.computeServer.get(
server.parentServer.id
)
String devicePath = exportSnapshotToHost(
hypervisor, snapVolInfo
)
snapshotFile.exportPath = devicePath
}
snapshot.addToSnapshotFiles(snapshotFile)
}
return ServiceResponse.success(snapshot)
}
NOTE: The snapshot.externalId you return is stored by the UI on each SnapshotFile record’s config (under a snapshot key). When revertSnapshot() or removeSnapshot() is called later, the Snapshot object passed back to your plugin will have this externalId populated from that stored config. This is how the UI tracks which array-side snapshot set corresponds to which Morpheus snapshot across multiple datastore types.
NOTE: When forExport is true, the exportPath on each SnapshotFile should be a device path or file path accessible on the hypervisor that the Morpheus agent (morphd) can read to stream the snapshot data to a backup target. For block storage, this typically means mapping/exporting the snapshot volume to the hypervisor and discovering its multipath device, similar to what prepareHostForVolume() does during provisioning.
Reverting a Snapshot
When a user reverts a snapshot, the HVM provisioner automatically powers off the VM before calling your revertSnapshot() method and powers it back on afterward. Your implementation only needs to handle the storage-side revert.
The Snapshot passed to this method will have its externalId populated (the array-side snapshot set ID you returned during creation) and its snapshotFiles list populated with the SnapshotFile entries your plugin created.
@Override
ServiceResponse<Snapshot> revertSnapshot(
ComputeServer server, Snapshot snapshot) {
if (!snapshot.externalId) {
return ServiceResponse.error(
'Snapshot has no external reference for revert'
)
}
// 1. Revert the snapshot set on the storage array
def revertResult = sdsClient.revertSnapshotSet(
snapshot.externalId
)
if (!revertResult.success) {
return ServiceResponse.error(
"Failed to revert snapshot: ${revertResult.error}"
)
}
// 2. After revert, the underlying block devices may have
// changed (e.g., new multipath device paths). Re-discover
// the device paths and update the volume records.
server.volumes?.each { vol ->
if (vol.datastore?.type?.code == getCode()) {
ComputeServer hypervisor =
morpheusContext.services.computeServer.get(
server.parentServer.id
)
String devicePath = discoverMultipathDevice(
hypervisor, vol
)
vol.externalId = devicePath
vol.setConfigProperty('source.dev', devicePath)
morpheusContext.services.storageVolume.save(vol)
}
}
return ServiceResponse.success(snapshot)
}
NOTE: After a revert, many storage arrays reassign device identifiers or refresh multipath mappings. It is important to re-discover the device paths for each volume and update both externalId and the source.dev config property so that the VM can be restarted with correct libvirt XML. The HVM provisioner will call applyDiskConfig() and redefine the VM after your revert completes.
Removing a Snapshot
Snapshot removal should delete the snapshot on the storage array. If the snapshot was previously exported for backup (i.e., forExport was true during creation), you should also clean up any exported multipath devices on the cluster hosts.
@Override
ServiceResponse removeSnapshot(
ComputeServer server, Snapshot snapshot) {
if (!snapshot.externalId) {
return ServiceResponse.error(
'Snapshot has no external reference for removal'
)
}
// 1. Check if any snapshot files were exported (have
// exportPath set) and clean up host-side devices
snapshot.snapshotFiles?.each { snapshotFile ->
if (snapshotFile.exportPath) {
cleanupExportedDevice(server, snapshotFile)
}
}
// 2. Delete the snapshot set on the storage array
def deleteResult = sdsClient.deleteSnapshotSet(
snapshot.externalId
)
if (!deleteResult.success) {
return ServiceResponse.error(
"Failed to delete snapshot: ${deleteResult.error}"
)
}
return ServiceResponse.success()
}
NOTE: The UI handles removing the Snapshot and SnapshotFile domain records from the database after your plugin returns success. You do not need to call morpheusContext.services.snapshot.remove() yourself during removeSnapshot().
Snapshot Lifecycle Summary
The following table summarizes the division of work between the Morpheus UI and your plugin for each snapshot operation:
| Operation | UI Responsibility | Plugin Responsibility |
|---|---|---|
Create |
Creates parent |
Creates snapshots on storage array, builds |
Revert |
Powers off VM, calls plugin, re-applies disk config, redefines VM, powers on |
Reverts snapshot set on storage array, re-discovers device paths, updates volume records |
Remove |
Deletes |
Cleans up exported devices (if any), deletes snapshot set on storage array |
Delegating to Built-in QCOW2 Datastore Operations
If your custom datastore type wraps a filesystem-based mount (e.g., an SDS solution that exposes a POSIX filesystem or an NFS pool backed by your storage array), you do not need to reimplement the standard QCOW2 volume operations (create, clone, resize, remove). Morpheus provides the MorpheusVmeQcow2DatastoreService that exposes the same directory-based volume logic used by the built-in libvirt-dir and libvirt-netfs-nfs datastore types.
Access this service via:
MorpheusVmeQcow2DatastoreService qcowService =
morpheusContext.services.storage.vmeQcow2DatastoreService
The service provides the following operations:
-
createVolume(StorageVolume volume, ComputeServer server)— Creates a QCOW2 file on the datastore’sexternalPath -
cloneVolume(StorageVolume volume, ComputeServer server, StorageVolume sourceVolume)— Clones a source QCOW2 to a new volume usingcporqemu-img -
cloneVolume(StorageVolume volume, ComputeServer server, VirtualImage virtualImage, CloudFileInterface cloudFile)— Streams a virtual image file directly to the datastore -
resizeVolume(StorageVolume volume, ComputeServer server, Long newSize)— Resizes a QCOW2 file viaqemu-img resize -
removeVolume(StorageVolume volume, ComputeServer server, boolean removeSnapshots, boolean force)— Deletes the QCOW2 file and its directory
This is useful when your DatastoreTypeProvider needs to handle the datastore lifecycle itself (create/remove datastore, register hosts, manage mounts) but wants the standard QCOW2 behavior for individual volume operations. For example, an SDS plugin that provides a shared filesystem datastore can delegate all volume methods:
class MySdsFilesystemDatastoreProvider implements
DatastoreTypeProvider {
// Datastore lifecycle -- custom (mount the filesystem,
// register hosts, etc.)
@Override
ServiceResponse<Datastore> createDatastore(Datastore ds) {
// Mount the SDS filesystem on each cluster host
// Set ds.externalPath to the mount point
}
// Volume operations -- delegate to built-in QCOW2 service
@Override
ServiceResponse<StorageVolume> createVolume(
StorageVolume volume, ComputeServer server) {
return morpheusContext.services.storage
.vmeQcow2DatastoreService
.createVolume(volume, server)
}
@Override
ServiceResponse<StorageVolume> cloneVolume(
StorageVolume volume, ComputeServer server,
StorageVolume sourceVolume) {
return morpheusContext.services.storage
.vmeQcow2DatastoreService
.cloneVolume(volume, server, sourceVolume)
}
@Override
ServiceResponse<StorageVolume> cloneVolume(
StorageVolume volume, ComputeServer server,
VirtualImage virtualImage,
CloudFileInterface cloudFile) {
return morpheusContext.services.storage
.vmeQcow2DatastoreService
.cloneVolume(volume, server,
virtualImage, cloudFile)
}
@Override
ServiceResponse<StorageVolume> resizeVolume(
StorageVolume volume, ComputeServer server,
Long newSize) {
return morpheusContext.services.storage
.vmeQcow2DatastoreService
.resizeVolume(volume, server, newSize)
}
@Override
ServiceResponse removeVolume(
StorageVolume volume, ComputeServer server,
boolean removeSnapshots, boolean force) {
return morpheusContext.services.storage
.vmeQcow2DatastoreService
.removeVolume(volume, server,
removeSnapshots, force)
}
// ... datastore lifecycle methods ...
}
NOTE: The MorpheusVmeQcow2DatastoreService operates on the volume’s datastore.externalPath and expects a standard directory layout. Ensure your datastore’s externalPath is set to the mount point of your filesystem before delegating volume operations.
Additionally, morpheusContext.services.storage.datastoreType provides MorpheusDatastoreTypeService — a routing service that dispatches volume and snapshot operations to the correct datastore provider based on the volume’s associated datastore type. This is useful when your plugin needs to invoke operations on volumes that may reside on a different datastore type (e.g., triggering a clone on a directory-based datastore that holds the source image cache for your block-based datastore).
Cluster Packages for Host Software
Storage plugins often require specific software to be installed on hypervisor hosts — for example, iSCSI initiator utilities, multipath tools, or a custom SDS client agent. Morpheus provides a cluster-level package system through the ComputeTypePackageProvider interface that allows plugins to define installable packages for HVM clusters.
A ComputeTypePackageProvider registers a ComputeTypePackage that appears as an addon package available for HVM clusters. When installed, Morpheus calls your provider’s installPackage() method with the cluster, where you can execute the necessary installation commands on each host.
class MySdsHostPackageProvider implements
ComputeTypePackageProvider {
MorpheusContext morpheusContext
Plugin plugin
MySdsHostPackageProvider(Plugin plugin,
MorpheusContext morpheusContext) {
this.morpheusContext = morpheusContext
this.plugin = plugin
}
@Override
String getCode() { return 'my-sds-host-package' }
@Override
String getName() { return 'My SDS Host Agent' }
@Override
String getDescription() {
return 'Installs SDS client tools on cluster hosts'
}
@Override
String getType() { return 'my-sds-agent' }
@Override
String getPackageType() { return 'storage' }
@Override
String getProviderType() { return 'mvm' }
@Override
String getPackageVersion() { return '1.0.0' }
@Override
Icon getCircularIcon() {
return new Icon(path: 'my-sds-icon.svg',
darkPath: 'my-sds-icon-dark.svg')
}
@Override
List<OptionType> getOptionTypes() {
// Optional: configuration options for the package
// (e.g., SDS cluster endpoint, auth token)
return []
}
@Override
ServiceResponse<ComputeServerGroupPackage> installPackage(
ComputeServerGroup serverGroup,
ComputeServerGroupPackage pkg,
PackageInstallRequest request) {
// Get all hosts in the cluster
List<ComputeServer> hosts =
morpheusContext.services.computeServer.list(
new DataQuery().withFilter(
'serverGroup.id', serverGroup.id
).withFilter(
'computeServerType.nodeType', 'morpheus-kvm-node'
)
)
// Install the SDS client on each host
hosts.each { host ->
def result =
morpheusContext.executeCommandOnServer(
host,
'apt-get install -y my-sds-client ' +
'multipath-tools open-iscsi'
).blockingGet()
if (!result.success) {
return ServiceResponse.error(
"Install failed on ${host.name}: " +
"${result.data}"
)
}
}
pkg.installed = true
pkg.status = 'ok'
return ServiceResponse.success(pkg)
}
@Override
ServiceResponse deletePackage(
ComputeServerGroup serverGroup,
ComputeServerGroupPackage pkg,
PackageDeleteRequest request) {
// Uninstall the SDS client from each host
return ServiceResponse.success()
}
}
Register the package provider in your plugin class alongside your other providers:
class MyStoragePlugin extends Plugin {
@Override
void initialize() {
// ... other providers ...
this.registerProvider(
new MySdsHostPackageProvider(this, this.morpheus)
)
}
}
The ComputeTypePackage.ProviderType enum includes MVM for HVM clusters. Once registered, the package appears in the cluster’s addon packages UI, where administrators can install, upgrade, or remove it. The package tracks its state via ComputeServerGroupPackage (with fields for installed, status, packageVersion, errorMessage).
NOTE: The package system operates at the cluster level, not the individual host level. Your installPackage() implementation should iterate over all hosts in the cluster. When new workers are added to the cluster, you can handle software installation via the EventSubscriberFacet ADD_WORKER event on your DatastoreTypeProvider as well, ensuring new hosts are configured correctly.
NOTE: For simpler cases where you just need to ensure a few OS packages are present on a host, you can also handle this imperatively in your createDatastore() or prepareHostForVolume() methods using morpheusContext.executeCommandOnServer() without implementing a full ComputeTypePackageProvider. The package provider approach is better suited for complex installations that benefit from explicit lifecycle management (install, upgrade, remove) and visibility in the UI.
Key Model Objects
Understanding the key Morpheus model objects is essential:
-
StorageServer— Represents the storage array integration. Has aconfigMapfor storing array-specific metadata (credentials endpoint, serial number, feature flags). Linked to aStorageServerTypedefined by yourStorageProvider. -
Datastore— A storage target on an HVM cluster. HasrefIdpointing to the cluster,storageServerlinking to the array,configMapfor datastore-specific config (host set name, protocol type), andexternalPathfor directory-based datastores. -
DatastoreLocation— Tracks per-hypervisor status of a datastore in a cluster. One entry per hypervisor host, used for tracking provisioning status and host-specific configuration. -
StorageVolume— A virtual disk. HasmaxStorage(size in bytes),externalId(device path or file path),volumeName,datastorereference,configMapfor volume metadata (WWN, UID, host mappings). TheconfigMapcan includesource.devfor the block device path used in libvirt XML. -
StorageVolumeType— Defines a type of volume your plugin offers (e.g., "SDS Block Volume", "SDS Replicated Volume"). -
ComputeServer— The VM/workload server. Useserver.parentServerto get the hypervisor host. Useserver.serverGroupor the datastore’srefIdto get the cluster. -
Snapshot— Represents a point-in-time snapshot. Hasname,externalId(your array-side snapshot set ID),snapshotFiles(list ofSnapshotFile),server,currentlyActive,parentSnapshot(for snapshot chains). Your plugin builds this as a model object and returns it; the UI persists it. -
SnapshotFile— Represents a single volume’s snapshot within aSnapshot. Hasname,externalId(per-volume snapshot ID),type(e.g.,block),volume(theStorageVolumeit was snapshotted from),exportPath(device path for backup agent access),diskIndex, andsnapshot(parent reference).
Concrete Example: Ceph RBD Storage for HVM
This section uses the built-in Ceph RBD implementation in Morpheus as a concrete reference for how the patterns described in this guide are applied in a real-world SDS solution. Ceph is a distributed storage system that provides both a POSIX filesystem (CephFS) and block storage (RBD). Its built-in implementation in Morpheus is not a plugin (it is part of the core HVM cloud code), but the architecture maps directly to what a plugin developer would build. Understanding this example will help you design your own SDS storage plugin.
Storage Architecture: CephFS + RBD
A Ceph-backed HVM cluster uses two complementary datastore types:
-
CephFS directory datastore (
libvirt-dir) — A CephFS mount on each hypervisor (e.g.,/mnt/cephfs/morpheus-images) appears as a standard directory-based datastore. This stores the shared QCOW2 virtual image cache. CephFS does not have a dedicated datastore type in Morpheus — it is simply discovered as alibvirt-dirdatastore because it presents a POSIX directory path. -
RBD pool datastore (
libvirt-ceph-rdb) — An RBD pool (e.g.,morpheus-kvm) registered as a libvirt storage pool on each hypervisor. This is where VM block volumes are created as RBD images. The datastore type code islibvirt-ceph-rdb(note the historicalrdbtypo that persists for backward compatibility).
Additionally, a special mvm-image-cache RBD pool is used as an intermediate cache for raw-format images. Virtual images are first downloaded as QCOW2 to the CephFS directory datastore, then converted to raw format in the mvm-image-cache RBD pool. VM volumes are then instant-cloned from protected snapshots in this cache pool.
This three-tier architecture (CephFS for QCOW2 store → RBD cache for raw images → RBD pool for VM volumes) demonstrates the "Why You Need Both" principle from earlier in this guide.
Image Caching Flow
When a VM is provisioned from a virtual image on an RBD datastore, the following flow occurs:
1. QCOW2 source image exists on CephFS directory datastore
e.g., /mnt/cephfs/morpheus-images/morpheus-{imageId}
2. Check if raw cached image exists in mvm-image-cache RBD pool:
rbd info mvm-image-cache/morpheus-{imageId}
3. If NOT cached, convert and cache:
a. Get source image size:
qemu-img info /mnt/cephfs/.../morpheus-{imageId}
b. Create RBD image at source size:
rbd create --size {sizeMB} mvm-image-cache/morpheus-{imageId}
c. Convert QCOW2 to raw into the RBD image:
qemu-img convert -O raw -f qcow2 \
/mnt/cephfs/.../morpheus-{imageId} \
rbd:mvm-image-cache/morpheus-{imageId} \
--target-is-zero -n -W -m16
d. Create and protect a snapshot for COW cloning:
rbd snap create mvm-image-cache/morpheus-{imageId}@image
rbd snap protect mvm-image-cache/morpheus-{imageId}@image
4. Clone from the cached snapshot (instant COW clone):
rbd clone mvm-image-cache/morpheus-{imageId}@image \
{vmPool}/{volumeName}
5. Refresh the libvirt pool and resize to requested size:
virsh pool-refresh {vmPool}
virsh vol-resize {volumeName} --pool {vmPool} \
--capacity {requestedBytes}
This caching strategy means that only the first VM provisioned from a given image incurs the QCOW2-to-raw conversion cost. Subsequent provisions from the same image are near-instantaneous COW clones.
The corresponding code from the built-in CephDatastoreService.cloneVolume() method illustrates the pattern:
// Simplified from CephDatastoreService.cloneVolume()
String sourceVolume = "morpheus-${volume.sourceImage}"
String imagesPath = kvmProvisionService.getImagesPath(
hypervisor.serverGroup
)
String command = """
sudo rbd info "mvm-image-cache/${sourceVolume}"
EXISTS=\$?
if [ "\$EXISTS" != "0" ] ; then
SIZE="\$(sudo qemu-img info \
"${imagesPath}/${sourceVolume}" \
| awk 'NR==3' | cut -d "(" -f2 \
| cut -d ")" -f1 | awk '{print \$1}')"
SIZEMB=`expr \$SIZE / 1048567`
sudo rbd create --size \$SIZEMB \
"mvm-image-cache/${sourceVolume}"
sudo qemu-img convert -O raw -f qcow2 \
"${imagesPath}/${sourceVolume}" \
"rbd:mvm-image-cache/${sourceVolume}" \
--target-is-zero -n -W -m16
sudo rbd snap create \
"mvm-image-cache/${sourceVolume}@image"
sudo rbd snap protect \
"mvm-image-cache/${sourceVolume}@image"
fi
sudo rbd clone \
"mvm-image-cache/${sourceVolume}@image" \
"${volume.datastore.externalId}/${volume.volumeName}"
sudo virsh pool-refresh \
"${volume.datastore.externalId}"
sudo virsh vol-resize "${volume.volumeName}" \
--pool "${volume.datastore.externalId}" \
--capacity ${volume.maxStorage}
"""
NOTE: The qemu-img convert flags are significant: -W enables out-of-order writes for parallelism, -m16 uses 16 coroutines, --target-is-zero skips writing zeroes (the RBD image was just created), and -n skips target creation (already created via rbd create). These flags are critical for performance when converting large images.
Volume-to-Volume Cloning (RBD to RBD)
When cloning from a source volume that is already on an RBD datastore (e.g., cloning a VM), the code avoids the QCOW2 conversion entirely and uses RBD-native snapshot and clone operations:
// Simplified from CephDatastoreService.cloneVolume()
// when source is already an RBD volume
String snapName = "${sourceVolume.datastore.externalId}/" +
"${sourceVolume.externalId}@clone-${new Date().time}"
String command = """
sudo rbd snap create "${snapName}"
sudo rbd snap protect "${snapName}"
sudo rbd clone "${snapName}" \
"${volume.datastore.externalId}/${volume.volumeName}"
sudo rbd flatten \
"${volume.datastore.externalId}/${volume.volumeName}"
sudo rbd snap unprotect "${snapName}"
sudo rbd snap rm "${snapName}"
sudo virsh pool-refresh \
"${volume.datastore.externalId}"
sudo virsh vol-resize "${volume.volumeName}" \
--pool "${volume.datastore.externalId}" \
--capacity ${volume.maxStorage}
"""
The sequence creates a temporary snapshot, clones from it, flattens the clone (making it independent), then removes the temporary snapshot. The flatten step ensures the new volume has no dependency on the source — important for VM independence. If the source volume already has a user snapshot, that snapshot is used directly instead of creating a temporary one.
Libvirt XML for RBD Volumes
RBD volumes use <disk type="network"> in the libvirt domain XML, which tells QEMU to connect directly to the Ceph cluster via the librbd library — no kernel RBD map is needed for VM disk I/O:
<disk type="network" device="disk">
<driver name="qemu" type="raw"
cache="writeback" discard="unmap"/>
<source protocol="rbd"
name="morpheus-kvm/vm-volume-001">
<auth username="admin">
<secret type="ceph"
uuid="a1b2c3d4-e5f6-7890-abcd-ef1234567890"/>
</auth>
<host name="ceph-mon-1.example.com" port="6789"/>
<host name="ceph-mon-2.example.com" port="6789"/>
<host name="ceph-mon-3.example.com" port="6789"/>
</source>
<target dev="vda" bus="virtio"/>
</disk>
Key differences from block and file disk types:
-
type="network"instead oftype="block"ortype="file"— QEMU connects over the network vialibrbd -
protocol="rbd"— identifies the Ceph RBD protocol -
cache="writeback"instead ofcache="none"— RBD benefits from writeback caching because the Ceph cluster provides its own data safety guarantees -
No
io="io_uring"— thelibrbddriver handles I/O internally -
<auth>with<secret type="ceph">— Ceph authentication is embedded in the XML, referencing a libvirt secret UUID that holds the CephX key -
Multiple
<host>entries — lists all Ceph monitor hosts for redundancy
The source name is formatted as {pool}/{volumeName}. The auth secret UUID and monitor hosts are pulled from the datastore’s per-host config (datastore.configMap.hostConfig["{serverId}"]).
Per-Host Configuration for HA
Each hypervisor in the cluster may have its own Ceph connection configuration. The Ceph implementation stores per-host config in the datastore’s configMap under hostConfig["{serverId}"]:
// Structure of datastore.configMap.hostConfig
// (populated during Ceph pool discovery/sync)
[
hostConfig: [
"42": [ // serverId of hypervisor A
source: [
name: "morpheus-kvm", // RBD pool name
host: [
[name: "ceph-mon-1.local", port: "6789"],
[name: "ceph-mon-2.local", port: "6789"]
],
auth: [
username: "admin",
secret: [
uuid: "a1b2c3d4-..."
]
]
]
],
"43": [ // serverId of hypervisor B
source: [
name: "morpheus-kvm",
host: [
[name: "ceph-mon-1.local", port: "6789"],
[name: "ceph-mon-3.local", port: "6789"]
],
auth: [
username: "admin",
secret: [
uuid: "b2c3d4e5-..."
]
]
]
]
]
]
When building the libvirt XML, the code reads the current hypervisor’s config from this map for the <source> attributes and auth, but collects monitor <host> entries from all hypervisors and deduplicates them. This ensures every VM has a complete list of Ceph monitors regardless of which host it is running on — critical for HA failover.
NOTE: This per-host config pattern is important for any storage plugin where different hosts may have different connection parameters (different auth secrets, different paths to the storage network, etc.). Store per-host config in datastore.configMap.hostConfig["{serverId}"] and aggregate cluster-wide settings when building libvirt XML.
Snapshots with RBD
Ceph RBD provides native snapshot support via rbd snap commands. The built-in implementation handles snapshots as follows:
Creating a snapshot:
// Simplified from CephDatastoreService.createSnapshot()
Snapshot snapshot = new Snapshot(
externalId: "snap-${new Date().time}"
)
for (volume in volumes) {
String command = "sudo rbd snap create " +
"${volume.datastore.externalId}/" +
"${volume.externalId}@${snapshot.externalId}"
// For running VMs, freeze the guest filesystem first
if (isRunning) {
command = "sudo virsh domfsfreeze " +
"\"${server.externalId}\"; " +
"${command}; " +
"sudo virsh domfsthaw " +
"\"${server.externalId}\""
}
def results = executeHypervisorCommand(
server.parentServer, command
)
if (results.success) {
snapshot.addToSnapshotFiles(
new SnapshotFile(
name: volume.deviceDisplayName,
type: 'rbd',
volume: volume,
path: "rbd:${volume.datastore.externalId}/" +
"${volume.externalId}" +
"@${snapshot.externalId}",
externalId: "${volume.datastore.externalId}/" +
"${volume.externalId}" +
"@${snapshot.externalId}",
diskIndex: volume.displayOrder
)
)
}
}
return ServiceResponse.success(snapshot)
Key points:
-
Running VM snapshots use
virsh domfsfreeze/domfsthaw(via the QEMU guest agent) to quiesce the filesystem before snapshotting. Without the guest agent, the snapshot of a running VM is crash-consistent only, and the Ceph implementation returns a user-friendly error recommending the VM be powered off. -
The
SnapshotFile.externalIdis the full RBD snapshot path (pool/image@snapname), which is self-contained and used directly byrevertSnapshot()andremoveSnapshot(). -
The
SnapshotFile.typeis set torbd(matching the volume’sdiskType).
Reverting a snapshot:
// Simplified from CephDatastoreService.revertSnapshot()
for (snapshotFile in snapshot.snapshotFiles) {
String command = "sudo rbd snap rollback " +
"${snapshotFile.externalId}"
executeHypervisorCommand(
server.parentServer, command
)
}
return ServiceResponse.success(snapshot)
RBD snap rollback atomically reverts the image to the snapshot state. The HVM provisioner handles powering off the VM before calling this method and powering it back on afterward.
Removing a snapshot:
// Simplified from CephDatastoreService.removeSnapshot()
for (snapshotFile in snapshot.snapshotFiles) {
String command = "sudo rbd snap rm " +
"${snapshotFile.externalId}"
executeHypervisorCommand(
server.parentServer, command
)
}
return ServiceResponse.success(snapshot)
Volume Removal
Ceph volume removal illustrates the cleanup sequence for an RBD-based volume:
// Simplified from CephDatastoreService.removeVolume()
String command = """
sudo rbd snap purge \
"${volume.datastore.externalId}/${volume.externalId}"
sudo virsh pool-refresh \
"${volume.datastore.externalId}"
sudo rbd device list --format json \
| jq -r '.[].device' \
| while read -r DEVICE; do \
sudo rbd device unmap "\$DEVICE"; done
sudo virsh vol-delete "${volume.externalId}" \
--pool "${volume.datastore.externalId}"
"""
The sequence is: purge all snapshots on the volume → refresh the libvirt pool → unmap any kernel-mapped RBD devices → delete the volume via virsh. The rbd device unmap step handles cleanup of any kernel RBD mappings that may exist from cross-datastore copy operations (kernel mappings are not used for normal VM I/O, which goes through QEMU’s librbd).
NOTE: When the VM’s parentServer (hypervisor) is not available (e.g., the hypervisor was already removed), the Ceph implementation falls back to executing the removal command on any available online hypervisor in the cluster. This is possible because RBD is a shared-nothing distributed system — any client with Ceph credentials can manage any volume. Your plugin should implement similar fallback logic if your storage array supports management from any host.
Applying Ceph Patterns to a Plugin
If you are building a plugin for a Ceph-like distributed block storage system, the built-in Ceph implementation maps to the plugin interfaces as follows:
| Built-in Ceph Code | Plugin Equivalent |
|---|---|
|
|
|
|
|
|
|
|
|
|
CephFS directory mount |
Use built-in |
|
|
Per-host |
Set up during |
For the diskType='rbd' case, your buildDiskConfig() should return:
@Override
ServiceResponse<MvmDiskConfig> buildDiskConfig(
ComputeServerGroup cluster, ComputeServer server,
StorageVolume volume) {
MvmDiskConfig config = new MvmDiskConfig()
config.deviceType = MvmDiskConfig.DeviceType.DISK
config.diskMode = MvmDiskConfig.DiskMode.VIRTIO
config.diskType = 'rbd'
return ServiceResponse.success(config)
}
The HVM provisioner will then use the datastore.configMap.hostConfig to build the full <disk type="network"> XML with Ceph auth and monitor hosts automatically.
Best Practices
-
Parallel host operations — When preparing volumes or rescanning hosts across a cluster, execute operations in parallel with a bounded thread pool and configurable timeouts.
-
Idempotent operations — Storage operations should be idempotent where possible. If a volume creation fails partway, retrying should not create duplicates.
-
Error handling with context — Always include relevant identifiers (volume name, host name, array serial) in error messages. Return
ServiceResponse.error()with descriptive messages rather than throwing exceptions. -
Volume metadata — Use
StorageVolume.configMapand helper config classes to store array-specific metadata (WWN, UID, host set associations). This metadata is essential for subsequent operations like resize, snapshot, and removal. -
Cleanup on failure — If a multi-step operation fails partway (e.g., volume created but host preparation failed), implement rollback logic to clean up partial state.
-
Minimum volume sizes — Some storage arrays enforce minimum volume sizes. Handle this in your
createVolume()by enforcing a floor (e.g., 256 MiB) and adjusting the requested size accordingly. -
Directory datastore for images — Ensure documentation and setup guides for your plugin instruct users to configure a shared directory-based datastore (NFS, GFS2, etc.) alongside your block-based datastore for QCOW2 image caching.
-
Config map for libvirt overrides — Set
source.dev,source.file, orsource.poolin the volume’sconfigMap(serialized as theconfigJSON field) to control how the libvirt XML<source>element is generated, rather than relying solely onexternalId.
Task Plugin
Add custom Tasks types to Morpheus. See a full example.
Setup
Tasks are useful components of your provisioning workflow. This plugin allows you to create custom Tasks
-
Create a new class that implements
com.morpheusdata.core.TaskProvider -
Create a new class that extends
com.morpheusdata.core.AbstractTaskService. This service defines methods for task execution in a variety of contexts, described below.
Options
OptionType is an easy way to create configuration for your new Task.
Simply provide a list of com.morpheusdata.model.OptionType to the TaskProvider.getOptionTypes method.
@Override
List<OptionType> getOptionTypes() {
OptionType optionType = new OptionType(
name: 'myTask',
code: 'myTaskText',
fieldName: 'myTask',
optionSource: true,
displayOrder: 0,
fieldLabel: 'Text to Reverse',
required: true,
inputType: OptionType.InputType.TEXT
)
return [optionType]
}
Task Contexts
A task can be run in one of three contexts:
-
None/Local (
executeLocalTask) -
Remote (
executeRemoteTask) -
Instance (
executeContainerTask,executeContainerTask)
Task Logo
A custom logo can be used in the Morpheus UI by defining the Icon object in the new TaskProvider.getIcon() interface method. Before this was simply a hard coded icon referenced by a code name. Both dark mode and light mode icons can be defined.
UI Extensions
Morpheus UI Extension Plugins provide a way to expand the capabilities of the Morpheus UI. Render custom content as a tab on an Instance, or even on a Server. Create a global injection component to the main layout footer for support/chat services. The possibilities are growing with each release and new functionality since 0.8.0 brings a lot. See a full tabs example.
Setup Instance Tabs
Create a new class that extends com.morpheusdata.core.AbstractInstanceTabProvider. When the Morpheus UI builds the Instance UI it calls the renderTemplate method.
Below is a simple example binding the Instance object to the template model.
@Override
HTMLResponse renderTemplate(Instance instance) {
ViewModel<Instance> model = new ViewModel<>()
model.object = instance
getRenderer().renderTemplate("hbs/instanceTab", model)
}
|
Tip
|
Use MorpheusContext.buildInstanceConfig to get more details about your Instance. See com.morpheusdata.model.TaskConfig
|
Handlebars is the default provided template engine.
To override this default, implement the com.morpheusdata.core.InstanceTabProvider interface and then write
your own getRenderer() method.
|
Tip
|
The HandlebersRenderer provides a helper called {{nonce}} for injecting into script tags
|
Setup Server Tabs
Create a new class that extends com.morpheusdata.core.AbstractServerTabProvider. When the Morpheus UI builds the Server Details UI, it calls the renderTemplate method.
Below is a simple example binding the Instance object to the template model.
@Override
HTMLResponse renderTemplate(ComputeServer server) {
ViewModel<ComputeServer> model = new ViewModel<>()
model.object = server
getRenderer().renderTemplate("hbs/serverTab", model)
}
|
Tip
|
Use MorpheusContext.buildComputeServerConfig to get more details about your Server. See com.morpheusdata.model.TaskConfig
|
Handlebars is the default provided template engine.
To override this default, implement the com.morpheusdata.core.ServerTabProvider interface and then write
your own getRenderer() method.
Templating
See the Views section and the documentation for your templating engine for specific syntax.
Security Policies
User Permissions
Before a template is rendered in the UI, the InstanceTabProvider.show method is called to determine if the current user can view the custom Instance Tab.
For example, you may wish to check that the current User has been granted the custom permission defined by your plugin.
@Override
Boolean show(Instance instance, User user, Account account) {
def show = true
plugin.permissions.each { Permission permission ->
if(user.permissions[permission.code] != permission.availableAccessTypes.last().toString()){
show = false
}
}
return show
}
Content Security Policy
If your custom UI needs to include external resources such as scripts, stylesheets, or frames,
you may need to customize the Morpheus Content-Security-Policy Header to allow those elements to be loaded in the browser.
@Override
TabContentSecurityPolicy getContentSecurityPolicy() {
def csp = new TabContentSecurityPolicy()
csp.scriptSrc = '*.jsdelivr.net'
csp.frameSrc = '*.digitalocean.com'
csp.imgSrc = '*.wikimedia.org'
csp.styleSrc = 'https: *.bootstrapcdn.com'
csp
}
You can also use the per request nonce token to set the attribute on the <script/> tags you may be injecting:
<script src="blah.js" nonce="{{nonce}}"/>