For this reference architecture, FMJ Systems operates as a Managed Service Provider offering private cloud infrastructure to client organizations. Rather than requiring clients to purchase, host, and maintain their own hardware, FMJ Systems provisions and manages the underlying infrastructure while clients retain control over their own virtual environments.

The MSP model creates a clear division of responsibility between FMJ Systems and its tenants:

This model allows tenants to consume cloud resources without the overhead of managing the infrastructure layer, while FMJ Systems maintains visibility and control over the platform as a whole.

FMJ systems would offer 2 approaches to billing depending on the nature of the client. The first option would be a Tiered Package approach, the second option would be a custom engagement with the client where FMJ Systems and the client work out how much resources would be required and for how much a month.

For clients with straightforward or predictable resource requirements, FMJ Systems offers fixed monthly packages. Each tier provides a defined allocation of compute, memory, storage within the clients isolated environment.

Tenants are able to spread the resources across as many VMs or containers as they would like. All packages include the base tenant environment:

For clients with more complex or unique requirements, FMJ Systems works directly with the client to scope a custom environment. This involves an initial consultation to determine compute, storage, networking, and service requirements, after which FMJ Systems and the client agree on a monthly rate reflecting the provisioned resources

Custom engagements are well-suited to customers who requires resources outside of the existing standard packages.

Unfortunately, due to Proxmox's lack of resource quota enforcement for CPU and RAM, virtual machines and containers will be provisioned by FMJ Systems. Without this control, a single tenant would be able to over provision virtual machines and containers taking up all shared CPU and RAM resources.

Creating a custom provisioning portal for tenants to create VMs and containers would be the best option in this case. Utilizing a database to keep track of per tenant resources and API calls to create VMs and CTs, it would be possible to check if tenants are over provisioning their allocated resources. this is something a production environment would benefit from.

Once FMJ Systems and the client have come to an agreement on what is required for the clients environment, the tenant onboarding process can begin. The following is required to setup a tenants environment:

After all of this has been implemented for the tenant, they are able to login an access their resources for configuring their resources. For a more detailed look at what exactly needs to be done for tenants to be onboarded, please take a look at Tenant Onboarding Process

In a production deployment, the process of onboarding tenants would be automated through the Proxmox REST API, allowing FMJ Systems to onboard new clients quickly and consistently without manual configuration steps. the API supports full control over VM and container creation, resource pool management, SDN Zone creation, and permission assignments. Router and switch configuration can be achieved through an SSH based scripting. Tenant networks could also be implemented through Proxmox Software Defined Networking to eliminate the need for SSH based scripting for the router and switch, as tenant networking would all be handled through Proxmox.

Billing implemented with a fixed monthly charge is well suited to a reference implementation, but a production implementation would benefit from usage based billing. The Proxmox REST API exposes CPU, RAM, and network utilization on a per VM basis. Implementing a way to poll the API for resource usage per VM then aggregating the data by tenant resource pool could be used to track usage for billing purposes. this would enable FMJ systems to bill customers based on usage rather than a fixed monthly cost, which would be more practical at scale

For the demonstration of this architecture, I have only implemented the base environment for each tenant with isolated compute, networking and storage on the tenant cluster. This includes a web server and VPN in their public subnet, and an Active Directory environment for Proxmox authentication located in their private subnet. The active directory environment can be swapped out to use the tenants existing identity provider for authentication, and a DNS server can be put in place. These are the names of the 3 tenants I have implemented