Back in the early 2000s, whenever a Developer asked for something, we [painfully] gave it to them. The painful part wasn't about them, it was simply about procurement.

What is Procurement 🤔

In tech, I need to buy something from a vendor. I either can work with a reseller who aggregates multiple vendors for me, or work directly with a vendor. Both have their advantages and disadvantages but it comes down to cost, speed, interoperability and support.

If a Dev needed a server, I'd call up one of my contacts and I'd request a quote that comprised of:

• The server itself (rack or tower)

• A specification for CPU and memory

• Disk configuration, RAID cards (and maybe some SAN-related network cards)

• Cables

If approved, I'd place the order and the gear would arrive in 2 weeks. TWO WEEKS. WHAT SPEED. Seriously, procurement times would vary and a lot of this would be tied to logistics, availability, and delivery. Once it all arrived, I still needed time to rack, and configure the server install an OS, patch it, add it to the network, join a domain, and do some final checks/tests for readiness. Once all was good, that Dev got access. Until they needed another server.

Kubernetes Namespaces 🪄

Within the development world, you have multiple different teams such as developers, QA testers, and many more. Each of these teams requires a dedicated environment to carry out their work. Provisioning them with dedicated clusters can become very expensive and is, therefore, not an ideal solution. Instead, why not provide them with dedicated namespaces within the same Kubernetes cluster? You can spin up a namespace very easily and provision it to the individual teams. However, if you had to manually create all the appropriate authorization mechanisms and isolation boundaries for policies, that would involve a lot of manual work. This is where OpenUnison shines.

Splitting the cluster among all these different teams using namespaces is also what we call Multi-Tenancy. Multi-Tenancy in the simplest terms can be understood as “single resource, multiple users”.

When Kubernetes started gaining traction about a decade ago, multi-tenancy wasn’t a common discussion point. However, during the first KubeCon in 2016, OpenUnison showcased a multi-tenant Kubernetes stack, highlighting the potential and importance of this feature.

Benefits of Multi-Tenancy

1. Cost Efficiency: Multi-tenancy reduces the number of control planes, cutting costs associated with hardware, management, and auditing.

2. Human Resource Management: Kubernetes expertise is expensive and in high demand. Multi-tenancy minimizes the need for a Kubernetes expert in every team, reducing human resource costs.

3. Operational Efficiency: Instead of upgrading numerous clusters and versions, multi-tenancy allows for a single upgrade process, significantly reducing the workload.

4. Self-Service Model: App owners gain control over their environments, while cluster owners are freed from managing individual app owner requests.

To enable our cluster to be multi-tenant, we use a concept called NameSpaces as a Service (NaaS). It essentially manages the entire lifecycle of provisioning the namespace, setting the correct security boundaries, and allowing the developers and other teams to access only their specific namespace.

Tremolo NaaS🔓

Namespace as a Service (NaaS) simplifies the creation and management of Kubernetes namespaces while integrating with identity providers to enforce roles and permissions.

Out of the box, NaaS provides two primary roles: admin and view.

1. Admin View: grants comprehensive control over the namespace, excluding changes to cluster-wide objects like resource quotas

2. View role: Offers read-only access to resources within the namespaces

Additional roles can be customized to meet specific needs, such as granting support staff the ability to delete pods within the Kubernetes dashboard.

How do you manage access groups within a Kubernetes namespace after its creation and association with identity provider groups, ensuring that the cluster admin is not responsible for overseeing access permissions?

To address this, OpenUnison established an approval group, with the requester as the initial member. This setup not only provides a self-service interface for namespace creation but also for managing access.

There are three models for managing access groups:

1. Internal Self-Service Groups: In this model, the cluster owner controls the groups and authorization. While this offers control, it also requires implementing audit policies and reporting, all of which are provided out of the box.

2. External Groups: This model integrates directly with external systems like Okta, Active Directory, or GitHub. It allows for seamless namespace management driven by your corporate identity system, eliminating the need for manual access approvals. The downside is that management relies entirely on the corporate identity system.

3. Hybrid Group: This model leverages the ease of onboarding from external systems while allowing quick internal access adjustments. Most customers start with internal management, switch to external, and eventually adopt the hybrid model for the best of both worlds.

All these processes can be fully automated, ensuring no manual intervention is needed for creating namespaces or managing access.

Implementing NaaS within OpenUnison 🫣

In our upcoming demonstration, we'll showcase the process of requesting a new namespace. Enterprise customers often require additional features, such as chargeback information, which can be easily integrated into the namespace request form.

OpenUnison supports various integrations, enabling the use of tools like Backstage or ServiceNow for interface interactions, instead of the default portal. This flexibility is a key advantage of our approach to multi-tenancy, which balances imperative and declarative methodologies.

Additionally, the system allows writing objects directly to the API or a Git repository, enhancing the flexibility of the namespace creation process.

We can create a Kubernetes object with two flexible approaches;

1. Directly writing to the API Server

2. Using GitOps for version control

To accommodate both methods, OpenUnison includes tools for each. When writing directly to the API server, the process is immediate and simple. However, if you prefer GitOps, you can set the "Write to request" option to true, enabling the accumulation of objects into a buffer during the workflow. Once complete, these objects are pushed to a Git repository using an SSH key.

To ensure seamless integration, we provide a task to wait for the GitOps controller to kick in, maintaining workflow continuity. This dual approach allows you to start with direct API interactions and transition to GitOps when ready, using the same workflows.

NOW come the configs and fun part!!! 😍

Namespace as a Service (NaaS) requires two key components for effective operation.

1. A relational database: It is essential for storing authorization information, synchronizing external identity information into buffer groups for RBAC binding, maintaining audit data, and managing state for asynchronous processes.

2. An SMTP service: It is necessary for sending notifications to relevant stakeholders when access requests are made. While we support SMTP out-of-the-box, enterprises often prefer email notifications despite the rise of platforms like Slack and Teams.

For ease of setup, we provide a MariaDB database for simple deployment and an SMTP Black Hole for handling email notifications without forwarding them, preventing log errors.

You can install the above components by following the Open Unison Documentation.

Deploying The NaaS Portal 🚀

Testing MariaDB

If you need a simple database implementation for testing:

<https://raw.githubusercontent.com/OpenUnison/kubeconeu/main/src/main/yaml/mariadb_k8s.yaml>

If you use this database, use the following configuration information.

Testing SMTP Server

If you don't have an SMTP server available, you can use the SMTP Blackhole to have a place to send email without forwarding it to any recipients:

kubectl create ns blackhole
kubectl create deployment blackhole --image=tremolosecurity/smtp-blackhole -n blackhole
kubectl expose deployment/blackhole --type=ClusterIP --port 1025 --target-port=1025 -n blackhole

If you use the blackhole smtp service, use the following configuration information:

Next, update your values.yaml, uncommenting the database and smtp sections of your values.yaml. As an example, the below will work with the testing database and SMTP server:

database:
  hibernate_dialect: org.hibernate.dialect.MariaDBDialect
  quartz_dialect: org.quartz.impl.jdbcjobstore.StdJDBCDelegate
  driver: org.mariadb.jdbc.Driver
  url: jdbc:mariadb://mariadb.mariadb.svc.cluster.local:3306/unison
  user: unison
  validation: SELECT 1

smtp:
  host: blackhole.blackhole.svc.cluster.local
  port: 1025
  user: "none"
  from: donotreply@domain.com
  tls: false

For enterprises, the system supports Kerberos authentication with SQL Server, allowing you to mount a Kerberos keytab into the namespace for authentication instead of using a password, highlighting the enterprise-level support provided.

With your configuration updated, the next step is to choose which management model to use. We'll be doing the internal groups model.

To deploy the local management self-service model, first enable internal groups by adding the following to your values.yaml:

openunison:
  non_secret_data:
    SHOW_PORTAL_ORGS: "true"
  role_attribute: portalGroups
  groups:
    areJson: "true"
  enable_provisioning: true
  use_standard_jit_workflow: false
  naas:
    groups:
      internal:
        enabled: true

We provide instructions for customizing OpenUnison's configuration, specifically focusing on the role_attribute setting. He suggests changing the attribute name from groups to portalGroups

Explanation

• The role_attribute configuration dictates how OpenUnison processes and displays role-related information.

• By changing this attribute, you're customizing how OpenUnison presents group information. Instead of showing raw group data, it will present a more refined view, displaying permissions and roles in the context of namespaces and clusters.

• This customization helps in better managing and displaying user permissions within the web interface, making it clearer which permissions apply to which namespaces and clusters.

  

We made some additional configurations which are:

1. ChangeareJson to true: This enables the JSON format for groups or roles, ensuring that data is processed or displayed in JSON format.

2. Setenable_provisioning to true: This activates the provisioning feature, allowing for the automated setup of resources or user accounts within OpenUnison.

   

   We also do the following:

   1. Addactive_mq_pvc: true: This line should be added to enable the use of a Persistent Volume Claim (PVC) for ActiveMQ’s data store, in addition to the existing setting for enable_activemq. This change is intended to address issues related to disk space and log management that arose from using a database for ActiveMQ’s data store.

   2. Explanation: ActiveMQ can operate in two modes for its data storage: using a database or a PVC. Initially, a database was used due to local deployment constraints, but this approach caused excessive log growth and potential disk space issues. Switching to a PVC helps mitigate these problems by providing more manageable and scalable storage.

Before deploying using ouctl, ensure that you have a password file for both MariaDB and your SMTP server.

Now we are ready to DEPLOY!!!! 😍

helm repo update

There are a few points to keep in mind. If you run into errors, these will be useful for debugging.

1. PVC Creation: The Persistent Volume Claim (PVC) is created from a StatefulSet template. If the PVC is missing, scale the StatefulSet down to zero and then back up to one to recreate the PVC.

2. Scaling StatefulSet: Ensure that only one replica is running to avoid complications. Multiple replicas can cause issues with PVCs and deployment stability.

3. Check Quota: Verify that you have enough available volume space and that you haven’t exceeded your quota. If necessary, provision additional quota.

4. Re-run ouctl: Once everything is set up correctly and quotas are verified, rerun the ouctl command to continue with the deployment process.

An additional deployment for ActiveMQ is set up and running, functioning as a message bus. The version currently deployed is not High Availability (HA) and is intended primarily for initial setup and testing. It is not suitable for production environments.

Now go ahead and log into your OpenUnison!

An option there called local deployment so that's where your tokens and your dashboard

Now go click on your account information in the upper left-hand corner

As the first person setting up the system, you automatically become the administrator. This is because it's common practice to grant administrative rights to the person who initiates the setup process. The system is designed to automate this assignment, ensuring that the initial setup user has the appropriate administrative access from the start.

Now let's create k8s namespace!

Click on “New Kubernetes Namespace”

You can give your namespace a unique name and create the namespace.

Now comes the Open Approvals & Reports section.

Under the "Reports" section, locate the "Users" button and search for your name.

Use the "Add to Cart" feature to manage workflows for this user, such as assigning roles or running pre-approvals.

Navigate to "Open Approvals" to review and approve requests. As an administrator, you can decide on requests and customize attributes, such as including chargeback information in the approval process.

In the Kubernetes Dashboard, we can see that a role and role binding has been created.

To approve a namespace creation request, you can use the following steps:

1. Request Access: Navigate to "Request Access" and find the option for "Local Deployment." Expand the menu and select "Administration" to view the administrators.

2. Approval Role: As the first approver and the person who requested the namespace, you have the authority to approve access requests. This role is tied to the business owner’s responsibilities.

3. Request on Behalf: You can request access on behalf of others, as long as they are logged in first. This system supports "just-in-time" access management.

The out-of-the-box Namespace as a Service workflow will generate a Namespace and RoleBindings, along with the groups needed to manage access. This is rarely enough to satisfy multitenancy requirements. You'll generally want to create at least a ResourceQuota, maybe NetworkPolicys, etc. There are several customization points where you can add your tasks.

Now create a new namespace:

Not every Namespace needs the side cars that's right, so the question is how do we collect that information and then how do we provision it into our cluster?

In Kubernetes environments, not all namespaces require sidecar containers. To manage this effectively, OpenUnison provides a method to collect and use information regarding the need for sidecars through custom attributes.

Custom Attributes

The form for onboarding a new namespace can have custom attributes added. These attributes can be as simple as a text box, or as complex as a drop-down with values loaded from an external API. Adding an attribute requires an update to the [openunison.naas.forms.new](<http://openunison.naas.forms.new/>)_namespace.additional_attributes section. For instance, if you wanted as users if they want to enable istio sidecar injection in your namespace, you could add:

openunison:
  naas:
    forms:
      new_namespace:
        additional_attributes:
        - name: istio_enabled
          # what will be displayed on the form
          displayName: Enable Istio Sidecars
          # validating regular expression
          regEx: ".*"
          # An error to provide the user if the regex fails
          regExFailedMsg: "Invalid istio configuration"
          # minimum number of characters
          minChars: 0
          # maximum number of characters
          maxChars: 0
          # should the value of this attribute be unique in OpenUnison
          unique: false
          # can by one of list, text, dynamicList
          type: list
          # if list, the values and labels in the form of DisplayLabel=value
          values:
          - Enabled=enabled
          - Disabled=disabled

Once the attribute is added to a form, it's available to workflows as both a request object with the name of the attribute and a user attribute. Attribute values are also available to approvers to provide additional context.

Namespace Labels and Annotations

When the namespace is provisioned, it can have custom labels and annotations. These annotations and labels can be generated from custom attributes, or created in a (workflow hook)[#workflow_hooks]. The value for an annotation or label must be preset in the request object. You add custom labels through name/value pairs in openunison.naas.workflows.new_namespace.namespace_request_labels and annotations through openunison.naas.workflows.new_namespace.namespace_request_annotations. For instance, to add the istio-injection label to a new namespace, you would add:

openunison:
  naas:
    workflows:
      new_namespace:
          namespace_request_labels:
            istio-injection: istio_enabled

Upgrading a Chart in OpenUnison 📈

To upgrade the chart without rerunning the entire OpenUnison control command, you can use Helm to apply the updates. Here’s how:

1. Run the Upgrade Command:

    helm upgrade <release-name> <chart-name> --namespace <namespace>
   

   • <release-name>: The name of the release you want to upgrade.

   • <chart-name>: The name of the Helm chart you are upgrading.

   • <namespace>: The Kubernetes namespace where the release is deployed.

LOOK "enabled Istio sidecars"

2. Developer Tools:

To integrate OpenUnison with other systems, such as Backstage, you can use developer tools to access API information:

1. Open Developer Tools in your browser and navigate to the "Config" section (typically located about three-quarters of the way down the menu).

2. Click on the "Response" tab to view the API response data. This data includes all metadata driven by the server-side, as the form is dynamically generated based on this metadata.

3. API Documentation: Access the API documentation via the Swagger docs on the OpenUnison website. This documentation provides detailed information on how to generate client code and interact with the APIs.

3. Customizing for Namespace Sidecars:

To update your cluster configuration to handle sidecar injection based on form data:

1. Update Namespace Labels and Annotations: Configure your Helm chart or deployment settings to include the logic for injecting sidecars based on the attributes collected from the form.

2. Integration and Automation: Use the provided API responses to automate form handling and resource provisioning, ensuring that your multi-tenant environment is configured according to the specified requirements.

• Add Labels: The configuration involves adding a label (istio_injection) that corresponds to the request attribute (istio_enabled) for sidecar injection. This label helps manage sidecar injection in namespaces based on user input.

• Annotations: While annotations are similar to labels, they are not necessary for this particular task but can be configured in the same manner if needed

  

  

  

  BOOM!!!! 🎊🎉

  You’ve achieved a significant milestone: automating the onboarding process for Istio-enabled applications.

OpenUnison provides hooks that allow the injection of code in the form of a workflow at various points during the process of onboarding a namespace. This helps protect against the Noisy Neighbour problem.

1. Workflow Hooks: The system offers hooks to inject code at different stages of the namespace onboarding process. This customization is documented in the "customizing the namespace onboarding process" section.

2. Resource Quotas: They will use these hooks to implement a resource quota as part of the namespace onboarding.

3. Multi-Tenancy with V Cluster: Referring to the Kubernetes and Enterprise Guide Third Edition, Marc mentions that in the V Cluster chapter, after creating a namespace, they deploy a V Cluster into the namespace using a workflow.

4. Example of Use: The workflows can be extended to handle more complex multi-tenancy setups, such as deploying V Clusters within namespaces.

To learn about all of the different ways in which you can customize the NaaS Onboarding process, check out the official documentation.

Let's create a resource quota with input on the form. Take the the sample yaml that's in there for the pre-run workflow.

apiVersion: openunison.tremolo.io/v1
kind: Workflow
metadata:
  name: create-resource-quota
  namespace: openunison
spec:
  description: creates a ResourceQuota object
  inList: false
  label: Creates a ResourceQuota
  orgId: internal-does-not-exist
  tasks: |-
      - taskType: customTask
        className: com.tremolosecurity.provisioning.tasks.CreateK8sObject
        params:
            targetName: $cluster$
            template: |-
                kind: ResourceQuota
                apiVersion: v1
                metadata:
                  name: resource-quota
                  namespace: $nameSpace$
                annotations:
                    tremolo.io/managedByGit: "$useGit$"
                spec:
                    hard:
                        cpu: "5"
                        memory: 10Gi
                        pods: 10
            srcType: yaml
            writeToRequest: "$useGit$"
            requestAttribute: git-secret-cluster-k8s-$nameSpace$
            path: /yaml/ns/$nameSpace$/resourcequotas/

We can dynamically manage the number of pods by user input during the namespace onboarding process.

To finalize the workflow creation, you need to ensure the following:

1. Syntax Validation: The Open Unison operator validates the syntax of your workflow. If there are any mistakes or incorrect configurations, you will receive an error when you submit it.

2. Post-Creation Steps:

   • Add the Attribute: First, you need to inform Open Unison to include the newly created attribute.

   • Invoke the Workflow: Then, you need to configure Open Unison to call the created workflow after the namespace is created.

     

     Let’s look at the process of adding a new attribute to the form and integrating it into the workflow:

     • Locate the Attribute Section: Find the section in the values.yaml file where attributes for the form are defined. Specifically, look for the existing istio_enabled attribute.

1. Duplicate and Modify: Duplicate the istio_enabled block and modify it to create a new attribute for the maximum number of pods:

   • Name: Set the name to quote_pods.

   • Display Name: Set the display name to "Requested Maximum Number of PODS."

   • Regex: Add a regular expression to ensure only numbers are accepted.

   • Failure Message: Customize the failure message to indicate the input must be a number.

   • Min and Max Characters: Set minimum and maximum character limits to restrict the input to valid numbers.

   • Type: Change the type to text and set values to an empty list.

2. Integrate Attribute into Workflow:

   • After defining the new attribute, update the workflow to handle this new input.

   • Use the post namespace create workflow hook from the documentation to ensure that after a namespace is created, the resource quota object is also created based on the user input.

Navigate to the dashboard to verify that the namespace has been created. Confirm the presence of a resource quota for the namespace, and ensure that there are 20 resources highlighted as inputted by the user.

Conclusion 🤓

In this blog, we learned about the concept of namespace as a service and how it can enable multi-tenancy within a Kubernetes cluster. We also saw the different ways in which we can use Open Unison for creating NaaS, and how we can create and approve permission for different users.

Overall, NaaS and multi-tenancy enable us to reduce cluster costs while maintaining a high level of security.

Thank you for reading!! 🤩

If you found this blog interesting, check out Part 1 and Part 2 of the live streams to hear the full conversation.😊

If you liked this blog, feel free to share it with your companions and stay updated with the latest developments in Cloud Native networking by following us on Twitter at EmpathyOps and checking out our website!