By Andrew Wojnarek
Director of Capacity and Performance Management (CPM)
(ATS Innovation Center, Malvern, PA)
IBM Cloud Private is a Kubernetes based container orchestration platform that empowers developers with common services for self-service deployment, monitoring, logging and security, as well as a portfolio of middleware, data and analytics. With IBM Cloud Private, development and systems engineering teams develop and manage microservices-based applications. You may be familiar with microservices already as IBM helped develop and evangelize the Service-Oriented Architecture (SOA) in the past. The idea, with both, allows developers to create applications that are decoupled which allows each piece to scale independently.
IBM Cloud Private is more than just a Kubernetes wrapper. The incredible value of this product is the entire suite of services, and the ecosystem of efficiency! You get an integrated environment for managing containers that includes the container orchestrator, a private image registry, a management console, and monitoring frameworks. Furthermore, you get industry class support and software that runs optimized on the most durable and highest performing hardware on the market (IBM Power).
It’s important to understand why an organization would use IBM Cloud Private (ICP), and to illustrate that we will provide three use cases: “Net New”, “Modernization” and “Cloud Protection/Agility.”
Building Net New applications on ICP is one of our primary use cases because it allows us to take one thing and work it through the entire cloud-first stack. What we mean by this, is when you have a Net New application, you’re able to start with a relatively clean slate, and really design things with a cloud-first, developer friendly architecture.
Organizations will either develop Net New, or they’ll start the process of modernizing an existing legacy application. This is a much harder task compared to Net New, but arguably just as important. Containerizing legacy workloads will happen and using ICP is the most logical choice. A lot of times the most critical, business impacting, customer facing workloads are the legacy workloads; so, putting your trust in a very stable, mature and supported platform makes sense.
Lastly “Cloud Protection/Agility” is our third use case. This is future proofing your workloads against vendor lock-in. The containerization of your workloads will ensure that you’ll be able to run your workloads anywhere in the world: on-prem, in a cloud, hybrid etc.
The Infrastructure Setup
Our infrastructure environment consists of:
- IBM Power 822
- 6 LPARs running IBM Cloud Private
- 10 CPUs with SMT/2/4/8
- IBM PowerVM
- 64GB Memory
- 100GB Flash Storage IBM V9000
Our application environment consists of:
- IBM Cloud Private 2.1.0.2
Core Concepts
Before we get into the installation and setup, it’s necessary to understand some core concepts of IBM Cloud Private. There are 6 types of nodes that potentially run in a cluster, all responsible for carrying out different types of work. This is an important thing to understand, because in a microservices architecture, we’re able to scale based on function.
* Boot Node – A boot or bootstrap node is used for running installation, configuration, node scaling, and cluster updates. Only one boot node is required for any cluster. You can use a single node for both master and boot.
* Master Node – A master node provides management services and controls the worker nodes in a cluster. Master nodes host processes that are responsible for resource allocation, state maintenance, scheduling, and monitoring.
* Worker Node – A worker node is a node that provides a containerized environment for running tasks. As demands increase, more worker nodes can easily be added to your cluster to improve performance and efficiency. A cluster can contain any number of worker nodes, but a minimum of one worker node is required.
** Proxy Node – A proxy node is a node that transmits external request to the services created inside your cluster.
Management Node – A management node is an optional node that only hosts management services such as monitoring, metering, and logging. By configuring dedicated management nodes, you can prevent the master node from becoming overloaded. You can enable the management node only during IBM Cloud Private installation.
VA Node – A VA (Vulnerability Advisor) node is an optional node that is used for running the Vulnerability Advisor services. Vulnerability Advisor services are resource intensive.
(Anything marked with * is required)
(Anything marked with ** is required for High Availability)
Installation and Setup
The installation and setup of IBM Cloud Private is a definitely selling point: setting up vanilla Kubernetes can be a daunting task – even for a seasoned Systems Engineer. There are a whole suite of software defined products that wrap open-source solutions:
These products do a great job at doing the following: understanding the strengths of the software and the gaps they have in the enterprise. For example – with Kubernetes: amazing software, incredible resilient but can be challenging to architect and manage. IBM (with IBM Cloud Private) has smoothed that process out and has provided a suite of tools and interfaces that fill those gaps – backed by industry leading support.
We’re not going to include any screenshots of the installation, because frankly it was too easy to even mention:
- Build the LPARs
- Install the pre-built IBM Cloud Private image
- Customize and Deploy the cluster
Now clearly there were steps in between, but we were so incredibly pleased with how easy it was to install and deploy this product. Being intimately familiar with OpenStack – this wasn’t even 1% of the difficulty of installing and configuring OpenStack.
Getting the cluster up and running was the easy part and testing the cluster for resiliency was the fun part. After the cluster was running, we went through and started taking nodes down to see how resilient things were. The problem with doing resiliency testing is you need something that can be impacted. It’s tough to pull power, or shut things down, when you don’t have a ‘state aware’ application where we can monitor how it was handled.
So, we wrote a little Golang app that’s very simple. You click a button, and it increments the counter (stored in a DB).
With this small containerized workload, we were able to set it up end to end with our new installation and test the resiliency of shutting down nodes within the cluster.
Running the Cluster
Remember when we said IBM has wrapped an open source software solution with an enterprise look and feel? Aside from the installation, this is where the product really shines.
IBM Cloud Private has a well-polished and enterprise grade management console that really excels at putting all of the pieces together. You can get a bird’s eye view of the health of the cluster right from the dashboard, view your deployments, and where they’re residing, and what they’re doing. You can deploy applications from the Helm catalog. Here are some of the things we thought were interesting:
GUI to CLI helper
So, on every screen you can click a button that shows what the command line equivalent would be to access the same information.
By no means necessary would we ever imply that using a GUI is going to be more efficient than administering a Kubernetes cluster on the command line – and neither is IBM. This feature is a great way to skill up on administrating Kubernetes through their suite of tools.
Spectrum Scale vs NFS
We architected this PoC to use NFS for /var/lib/registry. This filesystem is used for a shared registry store. If you’re unfamiliar with docker/kubernetes, this is where your images are stored so any master that dispatches deployments has a copy of the docker images, and the worker nodes can reach back out to that master and pull the images as needed.
In our opinion NFS is not a great production enterprise solution. There is a lot of protocol overhead, it does not recovery cleanly and the performance leaves something to be desired. We recommend using Spectrum Scale to provide the image registry – which would provide a robust and high performing way to delivery docker images.
Using Spectrum Scale will allow for faster recovery times of containers that die. Think about it this way if – or when – node fails, you’ll want a lower latency to recover and start your workloads back up on a healthy node.
Monitoring
The monitoring and metering of usage statistics is really well put together in this product. We were particularly pleased with how well everything is put together inside the management console.
The ability to see how many processors each workload has access to and the number of containers running at any given time is perfect for performance troubleshooting and capacity planning.
The three services we especially liked was Metering / Monitoring and Alerting. All access through Menu > Platform.
Sizing
One thing to consider is how to size the cluster. We went with 3 masters and 3 worker nodes. This is not ideal namely because we clearly do not want half of the cluster to be used for management purposes – we did this for PoC reasons. The reason why we chose to go with more master nodes, was because we wanted to test out high availability. The great thing about this cluster, is we could now expand it to many more worker nodes: x86, ppc64, s390 etc.
As of writing this PoC, the ICP dashboard says that the master nodes use roughly 46% of all memory – but little misleading since it reports on the memory used by the masters as well as the workers.
The IBM Power Hardware Advantage
Powerful forces—mobile, cloud and big data & analytics—are redefining how business gets done. Leaders are leveraging these forces to deepen relationships with customers and partners, drive new efficiencies and expand business models. IBM is the right partner to help you.
IBM Power Systems are designed for big data—from operational to computational to business and cognitive Watson solutions—are optimized for performance and can scale to support demanding and growing workloads. Capitalize on the currency of data by finding business insights faster and more efficiently. And gain the elasticity you need to handle the varying analytics initiatives your business requires.
The IBM Power 8/9 processors were designed with big data in mind. They’re truly remarkable, with features such as:
- Support for DDR3 and DDR4 memory through memory buffer chips that offload the memory support from the IBM POWER8 memory controller
- L4 cache within the memory buffer chip that reduces the memory latency for local access to memory behind the buffer chip; the operation of the L4 cache is transparent to applications running on the IBM POWER8 processor. Up to 128 MB of L4 cache can be available for each IBM POWER8 processor.
- Hardware transactional memory.
- On-chip accelerators, including on-chip encryption, compression, and random number generation accelerators.
- Coherent Accelerator Processor Interface (CAPI), which allow accelerators plugged into a PCIe slot to access the processor bus using a low latency, high-speed protocol interface.
- Adaptive IBM Power management.
As we referenced above in our benchmarking, another feature of the IBM Power 8 architecture is Simultaneous Multi-Threading (SMT). IBM Power 8 offers 8 threads per core. If your workload has throughput driven, SMT8 can offer incredible benefit.
Final Thoughts
Running your container workloads in the IBM Power ecosystem is something to definitely consider. This is hardware that’s optimized for big data, transactional workloads, and is on the forefront of innovation so companies can gain faster insights for competitive business advantages. The roadmap to getting legacy workloads into ICP is invaluable in itself – you’re containerizing for developer flexibility – but also to make everything generic. The ability to move and shift on demand is incredibly eye opening, for none other than the reason of pure price: on-prem to IBM Cloud to Amazon to Google and back.
Nobody is moving their entire organization to containers overnight. As we said before, what we’ve had best success with is to take one small workload and utilize design thinking. With a singular vision in mind, bring many flavors of IT together in one room to execute on that vision. The goal, in this case, will be to take that one workload:
- Containerize
- Decouple
- Orchestrate
- Test (resiliency and performance)
There is an IBM Cloud Private Community Edition, so there’s no barrier to entry. No huge implementation to pilot with a team of experts. This is cutting edge innovation, wrapped in industry leading enterprise features, backed by leading support.
Here at The ATS Group, we have a very diverse Innovation Center where we implement Proof of Concepts for customers looking to do this very exercise. We specialize in the implementation and architecture of advanced technology – we’d love to hear your story, and to talk to you about what problems you may have. Please contact us with any questions!
Since our founding in 2001, we have consulted on thousands of system implementations, upgrades, backups and recoveries. We also support customers by providing managed services, performance analysis and capacity planning. We are industry-certified professionals supporting SMBs, Fortune 500 companies, and government agencies. As experts in top technology vendors, we are experienced in virtualization, server and storage systems integration, containerized workloads, high performance computing (HPC), software defined infrastructure (SDI), devops, enterprise backup and other evolving technologies that operate mission-critical systems on premise, in the cloud, or in a hybrid environment.
Did this content resonate with you and your organization? Download the full version of the Proof of Concept: Build, Modernize and Protect with IBM Cloud Private to share with peers.