Exciting New Developments with unRAID 6! September 3, 2014Jon PanozzoBTRFS, cache pool, Company, Docker, Uncategorized, unRAID,Virtualization, Xen Hypervisor With the help of our community of loyal and dedicated beta testers, we have been hard at work on unRAID 6 since the beginning of this year and through it all, we’ve seen some exciting developments towards a bigger, better unRAID. For those of you who haven’t had time to keep up with the forum discussions, or who are new to unRAID, we wanted to summarize the key enhancements that are coming in this upcoming release:
64-bit architecture Cache pool ExtensionsSimplified manageability Improved application support
VirtualizationDocker containers Virtual machines (Xen & KVM)
New file systemsXFS BTRFS
64-bit ArchitectureMore and more we are seeing an industry trend putting primary application development focus on 64-bit architecture and many abandoning their 32-bit counterparts altogether. Since over 97 percent1 of our users report having a 64-bit capable system today, and over 60 percent1 of them have more than 4GB of RAM on their unRAID system, we upgraded to 64-bit to enable them to fully take advantage of their systems capabilities.
Cache Pool
One of the most popular features of unRAID 5 has been the ability to assign a storage device as a cache with the primary purpose of increasing write performance2. New in unRAID 6, you can group multiple storage devices of various sizes and types together into a cache pool.
The unRAID cache pool is created by using the BTRFS file system’s unique twist on traditional RAID 1. Here are just some of the benefits:
Improved Data Protection – With a single cache device, there’s a possibility that you can lose your data if the device fails before the data gets moved to the array. With a cache pool, however, all write operations are replicated across two separate disks to ensure that the loss of any one drive in the pool does not cause a system outage. Increased System Uptime – If a cache pool device fails, the system will continue to operate as normal. No need to drop everything to deal with a system outage. You can simply change the device when convenient. Better Scalability – Add more devices of different sizes to your pool as you need to and grow on-demand. Optimized for SSDs – With BTRFS, unRAID now has native support for TRIM, which can substantially reduce the number of write operations when used as a cache device. Benefits of SSDs vs. HDDs:They don’t require time to ‘spin up’ or consume a lot of power to operate (they are fast and efficient); They are also smaller, so you can fit more of them into a smaller space for highly compact, crazy fast storage. When used for storing large quantities of smaller files (e.g., metadata), SSDs can provide a faster response time for these files to the application compared to spinning hard disks; and SSDs are most ideal for supporting virtual machines. VM performance benefits on an SSD are comparable to what a user would experience with them on a desktop PC vs. a spinning disk.
Note: SSDs are only supported as part of a cache pool, not in the unRAID array at this time.
Optimized for Virtualization – Virtual machines and applications can have their data reside on the cache pool permanently for overall improved performance, while keeping mass-storage content on the array still accessible to those virtual instances at bare-metal I/O performance using VirtFS (for KVM Virtual Machines) and Docker (for Containers). Given the desire for “fast-as-you” responsiveness in application and machine performance, using the cache pool for virtual machine/application storage is a no-brainer. Use of SSDs in a cache pool extends this benefit even further.
By creating a cache pool, you can quickly capture data during the day. Because this device is not a part of the array, the write speed is not bottlenecked for parity calculations to occur. Then an unRAID process called “the Mover” copies the data from the cache to the array at time and frequency of your choosing. Once the Mover completes, the space consumed previously on the cache drive is once again freed up.
As you can see, cache pools are going to be a lot more valuable to users than just allowing for faster copies. They increase the performance, protection, and versatility of unRAID.
ExtensionsIn unRAID 5, all additional functionality that a user would want was achieved through the use of unRAID Plugins. While Plugins were incredibly valuable tools that allowed users to add significant new functionality to their servers, there were challenges relating to their ongoing support and development. In unRAID 6, Plugins are joined by Docker Containers and Virtual Machines to address those challenges and further simplify the overall user experience. unRAID 6 Extensions enable users to:
1. Enhance Core Functionality with Plugins. Plugins enable developers to manipulate the inner workings of the server OS to enable new functionality or increased levels of control over existing functionality. Plugins can be developed by both Lime Technology and community developers alike. Some of the most popular community developed plugins have been merged into the base OS from an original community work.
2. Run Powerful Applications with Docker Containers. Thanks to Docker containers, unRAID 6 can now run any almost any 64-bit Linux-based application. In addition, Docker’s approach to compartmentalizing applications and managing images allows for a “build-once, run-anywhere” approach to applications. This will greatly simplify the work of community developers. Docker is quite simply the perfect solution for supporting Linux applications on unRAID.
3. Discover New Use-Cases with Virtual Machines. VMs can greatly enhance the versatility of an unRAID system. Through the use of VMs, unRAID Server OS now has a place for use on the desktop PC and in the home theater. Stay tuned to the unRAID blog for more information on how we leverage host device “pass through” for some amazing capabilities!
unRAID 6 will include management tools for making use of all three of these types of extensions. Test out plugMan, xenMan, and dockerMan in the current beta to see for yourself how these Extension Managers will help you simplify manageability, safely support more applications, and run new and innovative virtual machines.
VirtualizationThere are three virtualization platforms supported in unRAID 6 that extend the capability of your system: Docker, KVM, and Xen. Each one of these platforms has a unique value proposition to unRAID. Let’s take a moment to review them.
Docker: Your solution for running Linux-based headless applications on unRAIDThe majority of applications that are available to install through traditional unRAID plugins can be run as Docker Containers today (including Plex, Dropbox, Crashplan, Owncloud, BT Sync, Transmission, etc.). In addition, there are countless others that were previously never available as an unRAID plugin (including Minecraft servers, web servers, and many more). Hop on over to the Docker Registry to search for more applications that you can run on your unRAID server. In addition, many of our own forum members have taken it upon themselves to build additional Dockerized applications as well a comprehensive Docker management tool for use in the unRAID 6 beta that’s available now. How does Docker benefit unRAID though? There are countless articles and videos that try to explain Docker, but as it applies to unRAID, the benefits are quite simple:
Better support for more applications; Improved application stability through isolation; Controlled runtime environments for Container authors; and Substantially reduced overhead when compared to virtual machines. The last point above is why Docker is our preferred platform for application virtualization.
In addition, there are still more applications available to unRAID in the form of Docker Containers than there are available through unRAID Plugins today. (In fact, there are currently 14,000 applications officially running on Docker today.)
The key difference between Docker Containers and Virtual Machines is in their approach to virtualization. Docker Containers leverage a technology concept called “Operating System Virtualization” whereas KVM and Xen act as “hypervisors” and emulate hardware, which is where the term virtual machines originated.
As the diagram above shows, Docker Containers use substantially less disk, CPU, and memory to run applications than any virtual-machine equivalent solution. VMs come with additional overhead that is brought on by their agnosticism of both the host hardware and the underlying guest operating system. This is where VMs differ from Containers. Containers only work with Linux-based operating systems, but that limitation is also their strength as they are optimized and aware of each other at the operating system, file system, and the kernel layers. It is ridiculously efficient at how it leverages BTRFS support for copy-on-write and snap-shotting to improve disk performance, endurance, and efficiency.
Virtual Machines: Xen and KVMDocker provides an ideal approach to running headless Linux-based applications on top of a storage appliance platform (NAS). VMs by themselves increase support for more applications (other operating systems). Advanced hardware virtualization capabilities like IOMMU (Intel VT-d / AMD-Vi) bring to light new potential use-cases for the unRAID OS itself, including home theater systems and desktop PCs.
VMs have some key advantages over Containers:
They support non-Linux operating systems (e.g. Windows) They provide drivers for underlying PCI devices independent of the host OS Not all applications are readily available as Containers just yet Expect to see more content from Lime Technology in the weeks/months ahead to demonstrate some of these capabilities and how unRAID seeks to change the way we look at how we store, protect, serve, and play digital content.
New File Systems
A major improvement we started adding in beta 6, and expanded on in beta 7, has been the support for new file systems in unRAID. The two new additions to unRAID 6 on top of the existing support for the ReiserFS file system are XFS and BTRFS. While the technical details of each can be researched in great depth online, here’s our view on how these file systems should be used in unRAID:
XFSSupports extremely large file systems (up to 8 exbibytes) Great for storing large data files (think media content) Fully mature and stable file system Ideally suited for unRAID array devices BTRFSSupports copy on write, snap-shot, and compression Allows mixing/matching different drives for use in an unRAID cache pool Ideally suited for use in the cache pool We have plans to support additional file systems as well, so stay tuned to our blog and forums for more updates as we progress in development!
Discuss this in the unRAID Community Forum (registration required to post)…
Notes: 1 Data from a poll we ran in the unRAID forum May 13-20, 2014. 2 The cache drive feature from previous unRAID versions always had one primary goal in mind: faster write performance. Generally speaking, users leveraging a disk array with 3 or more devices can see up to a 3x write performance improvement simply by adding a cache device to the system. By specifying a disk as a cache device, user shares could then be enabled to support its use. When data is written to a user share that has been configured to use the cache device, all of that data is initially written directly to the dedicated cache device. Because this device is not a part of the array, the write speed is not bottlenecked for parity calculations to occur. Then an unRAID process called “the mover” copies the data from the cache to the array at time and frequency of your choosing. Once the mover completes, the space consumed previously on the cache drive is once again freed to front-end other write operations to cache-enabled user shares.
64-bit architecture Cache pool ExtensionsSimplified manageability Improved application support
VirtualizationDocker containers Virtual machines (Xen & KVM)
New file systemsXFS BTRFS
64-bit ArchitectureMore and more we are seeing an industry trend putting primary application development focus on 64-bit architecture and many abandoning their 32-bit counterparts altogether. Since over 97 percent1 of our users report having a 64-bit capable system today, and over 60 percent1 of them have more than 4GB of RAM on their unRAID system, we upgraded to 64-bit to enable them to fully take advantage of their systems capabilities.
Cache Pool
One of the most popular features of unRAID 5 has been the ability to assign a storage device as a cache with the primary purpose of increasing write performance2. New in unRAID 6, you can group multiple storage devices of various sizes and types together into a cache pool.
The unRAID cache pool is created by using the BTRFS file system’s unique twist on traditional RAID 1. Here are just some of the benefits:
Improved Data Protection – With a single cache device, there’s a possibility that you can lose your data if the device fails before the data gets moved to the array. With a cache pool, however, all write operations are replicated across two separate disks to ensure that the loss of any one drive in the pool does not cause a system outage. Increased System Uptime – If a cache pool device fails, the system will continue to operate as normal. No need to drop everything to deal with a system outage. You can simply change the device when convenient. Better Scalability – Add more devices of different sizes to your pool as you need to and grow on-demand. Optimized for SSDs – With BTRFS, unRAID now has native support for TRIM, which can substantially reduce the number of write operations when used as a cache device. Benefits of SSDs vs. HDDs:They don’t require time to ‘spin up’ or consume a lot of power to operate (they are fast and efficient); They are also smaller, so you can fit more of them into a smaller space for highly compact, crazy fast storage. When used for storing large quantities of smaller files (e.g., metadata), SSDs can provide a faster response time for these files to the application compared to spinning hard disks; and SSDs are most ideal for supporting virtual machines. VM performance benefits on an SSD are comparable to what a user would experience with them on a desktop PC vs. a spinning disk.
Note: SSDs are only supported as part of a cache pool, not in the unRAID array at this time.
Optimized for Virtualization – Virtual machines and applications can have their data reside on the cache pool permanently for overall improved performance, while keeping mass-storage content on the array still accessible to those virtual instances at bare-metal I/O performance using VirtFS (for KVM Virtual Machines) and Docker (for Containers). Given the desire for “fast-as-you” responsiveness in application and machine performance, using the cache pool for virtual machine/application storage is a no-brainer. Use of SSDs in a cache pool extends this benefit even further.
By creating a cache pool, you can quickly capture data during the day. Because this device is not a part of the array, the write speed is not bottlenecked for parity calculations to occur. Then an unRAID process called “the Mover” copies the data from the cache to the array at time and frequency of your choosing. Once the Mover completes, the space consumed previously on the cache drive is once again freed up.
As you can see, cache pools are going to be a lot more valuable to users than just allowing for faster copies. They increase the performance, protection, and versatility of unRAID.
ExtensionsIn unRAID 5, all additional functionality that a user would want was achieved through the use of unRAID Plugins. While Plugins were incredibly valuable tools that allowed users to add significant new functionality to their servers, there were challenges relating to their ongoing support and development. In unRAID 6, Plugins are joined by Docker Containers and Virtual Machines to address those challenges and further simplify the overall user experience. unRAID 6 Extensions enable users to:
1. Enhance Core Functionality with Plugins. Plugins enable developers to manipulate the inner workings of the server OS to enable new functionality or increased levels of control over existing functionality. Plugins can be developed by both Lime Technology and community developers alike. Some of the most popular community developed plugins have been merged into the base OS from an original community work.
2. Run Powerful Applications with Docker Containers. Thanks to Docker containers, unRAID 6 can now run any almost any 64-bit Linux-based application. In addition, Docker’s approach to compartmentalizing applications and managing images allows for a “build-once, run-anywhere” approach to applications. This will greatly simplify the work of community developers. Docker is quite simply the perfect solution for supporting Linux applications on unRAID.
3. Discover New Use-Cases with Virtual Machines. VMs can greatly enhance the versatility of an unRAID system. Through the use of VMs, unRAID Server OS now has a place for use on the desktop PC and in the home theater. Stay tuned to the unRAID blog for more information on how we leverage host device “pass through” for some amazing capabilities!
unRAID 6 will include management tools for making use of all three of these types of extensions. Test out plugMan, xenMan, and dockerMan in the current beta to see for yourself how these Extension Managers will help you simplify manageability, safely support more applications, and run new and innovative virtual machines.
VirtualizationThere are three virtualization platforms supported in unRAID 6 that extend the capability of your system: Docker, KVM, and Xen. Each one of these platforms has a unique value proposition to unRAID. Let’s take a moment to review them.
Docker: Your solution for running Linux-based headless applications on unRAIDThe majority of applications that are available to install through traditional unRAID plugins can be run as Docker Containers today (including Plex, Dropbox, Crashplan, Owncloud, BT Sync, Transmission, etc.). In addition, there are countless others that were previously never available as an unRAID plugin (including Minecraft servers, web servers, and many more). Hop on over to the Docker Registry to search for more applications that you can run on your unRAID server. In addition, many of our own forum members have taken it upon themselves to build additional Dockerized applications as well a comprehensive Docker management tool for use in the unRAID 6 beta that’s available now. How does Docker benefit unRAID though? There are countless articles and videos that try to explain Docker, but as it applies to unRAID, the benefits are quite simple:
Better support for more applications; Improved application stability through isolation; Controlled runtime environments for Container authors; and Substantially reduced overhead when compared to virtual machines. The last point above is why Docker is our preferred platform for application virtualization.
In addition, there are still more applications available to unRAID in the form of Docker Containers than there are available through unRAID Plugins today. (In fact, there are currently 14,000 applications officially running on Docker today.)
The key difference between Docker Containers and Virtual Machines is in their approach to virtualization. Docker Containers leverage a technology concept called “Operating System Virtualization” whereas KVM and Xen act as “hypervisors” and emulate hardware, which is where the term virtual machines originated.
As the diagram above shows, Docker Containers use substantially less disk, CPU, and memory to run applications than any virtual-machine equivalent solution. VMs come with additional overhead that is brought on by their agnosticism of both the host hardware and the underlying guest operating system. This is where VMs differ from Containers. Containers only work with Linux-based operating systems, but that limitation is also their strength as they are optimized and aware of each other at the operating system, file system, and the kernel layers. It is ridiculously efficient at how it leverages BTRFS support for copy-on-write and snap-shotting to improve disk performance, endurance, and efficiency.
Virtual Machines: Xen and KVMDocker provides an ideal approach to running headless Linux-based applications on top of a storage appliance platform (NAS). VMs by themselves increase support for more applications (other operating systems). Advanced hardware virtualization capabilities like IOMMU (Intel VT-d / AMD-Vi) bring to light new potential use-cases for the unRAID OS itself, including home theater systems and desktop PCs.
VMs have some key advantages over Containers:
They support non-Linux operating systems (e.g. Windows) They provide drivers for underlying PCI devices independent of the host OS Not all applications are readily available as Containers just yet Expect to see more content from Lime Technology in the weeks/months ahead to demonstrate some of these capabilities and how unRAID seeks to change the way we look at how we store, protect, serve, and play digital content.
New File Systems
A major improvement we started adding in beta 6, and expanded on in beta 7, has been the support for new file systems in unRAID. The two new additions to unRAID 6 on top of the existing support for the ReiserFS file system are XFS and BTRFS. While the technical details of each can be researched in great depth online, here’s our view on how these file systems should be used in unRAID:
XFSSupports extremely large file systems (up to 8 exbibytes) Great for storing large data files (think media content) Fully mature and stable file system Ideally suited for unRAID array devices BTRFSSupports copy on write, snap-shot, and compression Allows mixing/matching different drives for use in an unRAID cache pool Ideally suited for use in the cache pool We have plans to support additional file systems as well, so stay tuned to our blog and forums for more updates as we progress in development!
Discuss this in the unRAID Community Forum (registration required to post)…
Notes: 1 Data from a poll we ran in the unRAID forum May 13-20, 2014. 2 The cache drive feature from previous unRAID versions always had one primary goal in mind: faster write performance. Generally speaking, users leveraging a disk array with 3 or more devices can see up to a 3x write performance improvement simply by adding a cache device to the system. By specifying a disk as a cache device, user shares could then be enabled to support its use. When data is written to a user share that has been configured to use the cache device, all of that data is initially written directly to the dedicated cache device. Because this device is not a part of the array, the write speed is not bottlenecked for parity calculations to occur. Then an unRAID process called “the mover” copies the data from the cache to the array at time and frequency of your choosing. Once the mover completes, the space consumed previously on the cache drive is once again freed to front-end other write operations to cache-enabled user shares.
Laatst bewerkt door een moderator:
