Production Deployment Guide

This comprehensive guide provides detailed instructions for deploying Ilum in production environments with enhanced security, namespace separation, and multiple configuration options to meet diverse operational requirements.

Overview
Kubernetes Prerequisites
Architecture
High Availability Configuration
Failure Domains & Resilience
Dynamic Resource Scaling
Namespace Separation Strategy
Security Configuration
Dependency Deployment
Pre-configured Stack Options
Helm Values Configuration
Installation Instructions
Post-Installation Configuration
Troubleshooting

Overview

For production environments, it's strongly recommended to deploy critical dependencies in separate namespaces to achieve:

Enhanced Security: Namespace-level isolation and RBAC policies
Resource Management: Independent resource quotas and limits
Operational Excellence: Simplified maintenance and upgrades
Compliance: Meeting organizational separation requirements
Scalability: Independent scaling of components

Critical Components for Namespace Separation

The following components should be deployed in separate namespaces for production:

MongoDB: Primary data storage
Apache Kafka: Message broker and communication layer
MinIO: Object storage for Spark applications and data
PostgreSQL: Metadata storage for various services

Kubernetes Prerequisites

Ilum has been extensively tested across all leading Kubernetes environments, ensuring compatibility with a variety of deployment scenarios:

Supported Platforms

Lightweight Distributions: k3s, Rancher, MicroK8s
Bare-metal Clusters: Self-managed Kubernetes installations
Managed Services:
- Google Kubernetes Engine (GKE)
- Amazon Elastic Kubernetes Service (EKS)
- Azure Kubernetes Service (AKS)
- DigitalOcean Kubernetes
- Red Hat OpenShift

Minimum Suggested Requirements

Component	Requirement
Kubernetes Version	1.20+
CPU	8 cores minimum, 16+ recommended
Memory	16GB minimum, 32GB+ recommended

Air-gapped (Offline) Environments

For air-gapped installations, refer to our comprehensive Air-gapped Installation Guide.

Testing vs Production

Minikube is used throughout our documentation for demonstration purposes but is not suitable for production due to limitations in scalability, resource management, and high availability.

Architecture

Components and modules

MongoDB

Ilum employs MongoDB as its storage layer, preserving all data required between restarts within the MongoDB database. Ilum automatically creates all necessary databases and collections during the startup process.

Apache Kafka

Apache Kafka serves as Ilum's communication layer, facilitating interaction between Ilum-Core and Spark jobs, as well as between different Ilum-Core instances when scaled. It is critical to ensure Apache Kafka brokers are accessible by both Ilum-Core and Spark jobs, especially when Spark jobs are launched on a different Kubernetes cluster.

Ilum utilizes Kafka to carry out communication using several topics, all created during Ilum's startup. Therefore, users don't need to manage these topics manually.

MinIO

Ilum uses MinIO as the storage layer for Spark application components. All files (including jars, configurations, data files) needed for the operation of Spark components (driver, executors) are stored and made available for download via MinIO.

MinIO implements the S3 interface, which also enables it to store input/output data.

PostgreSQL

PostgreSQL database is used by services such as Marquez, Hive Metastore, Airflow in order to store metadata.

PostgreSQL databases are enabled in Ilum by default.

If you want to control whether PostgreSQL is enabled or not, you can use helm value postgresql.enabled. For example, to disable it, you can add --set postgresql.enabled=false to your installation command.

Ilum-Livy-proxy

Ilum-Livy proxy is our implementation of Livy Api, that integrates spark code with Ilum Groups in services such as Jupyter, Zeppelin, Airflow

Ilum Livy-proxy is enabled in Ilum by default.

In case you want to add or remove Ilum-Livy-proxy, you can use ilum-livy-proxy.enabled helm value to manage it. For example: --set ilum-livy-proxy.enabled=false to disable it.

Jupyter

Jupyter is a Notebook - sophisticated development environment which allows you to have code, charts, explanations and more in one executable document.

Jupyter is enabled in Ilum by default.

However, in case you want to control whether it is enabled or not, you can use helm value ilum-jupyter.enabled. For example, you can add --set ilum-jupyter.enabled=false to your installation command to disable it.

Be aware, that Jupyter makes use of Ilum-Livy-proxy to integrate with Ilum Groups. Therefore, you should enable it as well: --set ilum-livy-proxy.enabled=true

If you want to access the Jupyter UI, you can do it by:

using Ilum UI: go to Modules > Jupyter
configuring an ingress
using the port-forward command kubectl port-forward svc/ilum-jupyter 8888:8888

Apache Zeppelin

Zeppelin is a Notebook - sophisticated development environment which allows you to have code, charts, explanations and more in one executable document.

Please be aware, that Zeppelin notebook is not bundled in ilum package by default. If you want to run this service, add --set ilum-zeppelin.enabled=true to your installation command.

Be aware, that Zeppelin makes use of Ilum-Livy-proxy to integrate with Ilum Groups. Therefore, you should enable it as well: --set ilum-livy-proxy.enabled=true

If you want to access the Zeppelin UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-zeppelin 8080:8080

Hive Metastore

Hive Metastore is a metadata storage used to store your Spark catalogs (Spark tables, databases, views, and more) in a database instead of runtime memory. You can view these schemas later on the Table Explorer page.

info

Hive Metastore is not enabled in Ilum by default.

To enable the Hive Metastore bundled instance, set the following values in your Helm installation command:

ilum-core:
  metastore:
    enabled: true
    type: hive

ilum-hive-metastore:
  enabled: true

note

Hive Metastore uses PostgreSQL database to store metadata. You can read about Postgres in Ilum below.

Project Nessie

Nessie is a transactional catalog for your data. It was inspired by Git and is designed to support a wide range of data-lake tooling. It works best with Apache Iceberg tables.

To learn more about Nessie, visit the Nessie documentation page.

info

Project Nessie is not enabled in Ilum by default. To enable it, set the following values in your Helm installation command:

ilum-core:
  metastore:
    enabled: true
    type: nessie

nessie:
  enabled: true

Ilum SQL

note

Ilum SQL is not enabled in Ilum by default.

To enable Ilum SQL, add --set ilum-sql.enabled=true to enable the SQL execution host and --set ilum-core.sql.enabled=true to enable the SQL featues inside Ilum itself.

Ilum SQL can execute SQL queries on your data in the UI. Read more about it on the SQL Viewer page.

Trino

Note: Ilum SQL is not enabled in Ilum by default. To enable it, add --set trino.enabled=true to enable a built-in Trino distribution.

Trino is a distributed SQL query engine that allows you to run queries quickly. It is substantially more useful than Spark when it comes to interactive queries.

Ilum uses Trino to run SQL queries on your data in the UI. More about it read on the SQL Viewer page.

n8n

Note: n8n is not enabled in Ilum by default. To enable it, add --set ilum-n8n.enabled=true to enable a built-in n8n distribution.

n8n is a fair-code workflow automation platform with native AI capabilities.

Apache Airflow

Apache Airflow is a powerful platform for orchestrating and managing data workflows. To read more about Airflow in Ilum, visit the Airflow documentation page.

note

Airflow is not enabled in the Ilum package by default. To deploy Airflow, add --set airflow.enabled=true to your installation command.

Once enabled, Airflow will appear in the Ilum UI under the Modules section.

Apache Livy

Airflow can leverage Ilum’s Livy proxy to easily create jobs within Ilum. For more details, see the Livy proxy section above.

Marquez

Marquez is an open-source metadata management tool that focuses on capturing, aggregating, and visualizing the lineage of data assets within an organization’s data ecosystem. It tracks how datasets are produced and consumed by different jobs and provides a central view of these dependencies.

Please be aware that Marquez is not bundled in Ilum package by default. If you want to run this service, add --set global.lineage.enabled=true to your installation command.

Take into account that Marquez makes use of PostgreSQL database to store the metadata. You can read about it below.

Additionally, if you wish to use Marquez’s web client instead of Ilum’s UI, enable the default web client with --set ilum-marquez.web.enabled=true and set up one of the access methods:

use the port-forward command kubectl port-forward svc/ilum-marquez-web 9444:9444
configure an ingress

Read more about Marquez and Ilum Lineage here

Kestra

Kestra is an open-source data orchestration platform designed for orchestrating and automating data pipelines and business workflows. You can read about it here.

Kestra is not enabled in Ilum by default. To enable it, add --set kestra.enabled=true to your installation command.

Kestra uses PostgreSQL database to store the data about the jobs and tasks and Minio for general file storage

Mage

Mage is an open-source data engineering platform that simplifies the process of building, deploying, and maintaining data pipelines. It provides a user-friendly interface for creating data workflows, integrating with various data sources, and managing data transformations.

To read about Mage next to Ilum, visit the documentation page.

info

Mage is not enabled in Ilum by default. To enable it, add --set mageai.enabled=true to your installation command.

note

Ilum deploys Mage OSS, which is the open-source version of Mage, and does not include the commercial features available in Mage Pro (Cloud).

NiFi

Apache NiFi is a software project for building data processing pipelines. It provides a user interface for creating, managing, and deploying data processing pipelines.

To read about NiFi next to Ilum, visit the documentation page.

info

NiFi is not enabled in Ilum by default. To enable it, add --set nifi.enabled=true to your installation command.

Streamlit

Streamlit is a library for creating beautiful, performant, and scalable data apps in Python. It is used to build custom data apps accessible from the Ilum UI.

To read about Streamlit next to Ilum, visit the documentation page.

info

Streamlit is not enabled in Ilum by default. To enable it, add --set streamlit.enabled=true to your installation command.

Additionally, can provide a docker image with your streamlit application running in it. To see how to do it, visit our documentation page.

Kube Prometheus Stack

Kube Prometheus Stack includes Prometheus, Grafana and other tools for monitoring your data infrastructure

Please be aware, that Kube Prometheus Stack is not bundled in ilum package by default. If you want to run this service, add --set kube-prometheus-stack.enabled=true to your installation command.

If you are upgrading an existing Ilum Helm chart that previously did not have the Kube Prometheus Stack enabled, you must first install the required Prometheus Custom Resource Definitions (CRDs) before proceeding with the upgrade. To do this, run the following commands:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusagents.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_scrapeconfigs.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.80.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml

If you want to access the Prometheus UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/prometheus-operated 9090:9090

If you want to access the Grafana UI, the best way to do it is by configuring an ingress or using the port-forward command kubectl port-forward svc/ilum-grafana 8080:80

Loki and Promtail

Loki is used to gather and manage logs of your data infrastructure. Promtail is used as an agent that pushes logs into Loki

Please be aware, that Loki is not enabled in Ilum by default. If you want to run this service, add --set global.logAggregation.loki.enabled=true to your installation command.

Promtail also is not enabled in Ilum by default. To enable it add --set global.logAggreagtion.promtail.enabled=true to your installation command

If you want to access Loki and run Loki Queries, you can configure an ingress or use the port-forward command kubectl port-forward svc/ilum-loki-read 3100:3100 for read queries and kubectl port-forward svc/ilum-loki-write 3100:3100 for write queries. You can also use service ilum-loki-gateway to link grafana to loki

Production Architecture Overview

Recommendations on which optionally deployed components to place in which namespace, whether in the one belonging to ilum release or in separate, dedicated ones

┌─────────────────────────┐     ┌────────────────────────────┐
│   ILUM                  │     │   Dependencies             │
│   components            │     │   separated                │
│   namespace             │     │   namespaces               │
├─────────────────────────┤     ├────────────────────────────┤
│ • Ilum Core **          │     │ • MongoDB **               │
│ • Ilum UI **            │     │ • Kafka **                 │
│ • Ilum Livy-Proxy *     │     │ • MinIO *                  │
│ • Ilum Jupyter *        │     │ • PostgreSQL *             │
│ • Ilum Zeppelin         │     │ • Kube-Prometheus-Stack *  │
│ • Ilum Hive Metastore * │     │ • Loki & Promtail *        │
│ • Ilum Kyuubi *         │     └────────────────────────────┘
│ • Airflow               │
│ • MlFlow                │
│ • Marquez *             │
│ • Superset              │
│ • Gitea *               │
│ • Trino                 │
│ • N8N                   │
│ • Kestra                │
│ • OpenLdap              │
└─────────────────────────┘
**  Mandatory components for production release
*   Recommended components for production release

Network Communication

Inter-namespace Communication: Configured via Kubernetes Services and NetworkPolicies
Service Discovery: DNS-based service resolution
Security: TLS encryption for all inter-service communication

High Availability Configuration

For production deployments, critical services should run with multiple replicas to eliminate single points of failure. The following table shows recommended configurations:

Component	Minimum Replicas	Recommended	Helm Value	Notes
ilum-core	2	3	`ilum-core.replicaCount: 3`	Stateless; requires Kafka communication mode for HA
MongoDB	3	3	`mongodb.replicaCount: 3`	Replica set with automatic failover
Apache Kafka	3	3	`kafka.replicaCount: 3`	Required for HA communication; minimum 3 for quorum
MinIO	4	4	`minio.replicas: 4`	Erasure coding requires minimum 4 nodes
PostgreSQL	2	3	`postgresql.replicaCount: 3`	Primary + standby with streaming replication
Hive Metastore	2	2	`ilum-hive-metastore.replicaCount: 2`	Stateless; connects to shared PostgreSQL backend

Example Helm values for a fully HA deployment:

ilum-core:
  replicaCount: 3

mongodb:
  replicaCount: 3

kafka:
  replicaCount: 3

minio:
  replicas: 4

postgresql:
  replicaCount: 3

warning

High Availability for ilum-core requires Apache Kafka as the communication type. The default gRPC communication does not support HA. Set ilum-core.communication.type: kafka to enable HA mode.

Failure Domains & Resilience

Ilum's architecture is designed to eliminate single points of failure through several mechanisms:

Stateless Core Design

The ilum-core component is fully stateless — it persists no local data and can recover its entire state from MongoDB and Kafka after a crash or restart. This means any ilum-core replica can handle any request, enabling seamless horizontal scaling and zero-downtime rolling updates.

Pod Anti-Affinity

Configure Kubernetes pod anti-affinity rules to ensure replicas of critical services are spread across different nodes and availability zones:

ilum-core:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values:
                    - ilum-core
            topologyKey: kubernetes.io/hostname

Namespace-Level Isolation

Deploying dependencies (MongoDB, Kafka, MinIO, PostgreSQL) in separate namespaces provides blast radius containment. A failure in one component's namespace does not cascade to others. See Namespace Separation Strategy for details.

Leader Election

For components that require coordination (such as the Spark History Server or scheduled maintenance tasks), Kubernetes-native leader election ensures that exactly one instance acts as the primary while others stand by for automatic failover.

Dynamic Resource Scaling

Ilum supports multiple layers of dynamic scaling to match compute resources with workload demands.

Spark Dynamic Allocation

Enable Spark's built-in dynamic allocation to automatically adjust the number of executors based on workload:

# Cluster-level Spark defaults in ilum
spark.dynamicAllocation.enabled: "true"
spark.dynamicAllocation.minExecutors: "1"
spark.dynamicAllocation.maxExecutors: "20"
spark.dynamicAllocation.executorIdleTimeout: "60s"
spark.dynamicAllocation.schedulerBacklogTimeout: "5s"

With dynamic allocation, idle executors are released back to the cluster, freeing resources for other jobs. New executors are added automatically when tasks queue up.

Kubernetes Horizontal Pod Autoscaler (HPA)

For always-on services like Trino coordinators or ilum-core, use Kubernetes HPA to scale based on CPU or memory utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ilum-core-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ilum-core
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Cluster Autoscaler Integration

Ilum works seamlessly with Kubernetes cluster autoscalers (available on GKE, EKS, AKS, and via Cluster API). When Spark jobs request more executor pods than the current cluster can accommodate, the cluster autoscaler provisions additional nodes automatically. When pods are released, nodes are drained and removed.

tip

Combine Spark dynamic allocation with cluster autoscaling for full elasticity: Spark scales executors based on workload, and the cluster scales nodes based on pending pods.

Auto-Pause on Idle

Ilum's interactive services (SQL Viewer sessions, Jupyter notebooks) can be configured to automatically pause after a period of inactivity, releasing compute resources:

ilum-core:
  sql:
    duckdb:
      idleTimeout: 1h
  job:
    retain:
      hours: 24

Namespace Separation Strategy

Recommended Namespace Structure

# Core Ilum Components placed in ilum namespace
ilum:          # Main application services

# Critical Dependencies (Separate Namespaces)
ilum-mongodb:       # MongoDB cluster
ilum-kafka:         # Kafka brokers and Zookeeper
ilum-minio:         # Object storage cluster
ilum-postgresql:    # PostgreSQL databases
ilum-kps:           # Kube-prometheus-stack - monitoring resources
ilum-logs:          # Loki & Promtail - log aggregation resources

Namespace Creation

Create all required namespaces before deployment:

⚠️ Warning: Create namespaces only for components you plan to deploy

# Ilum namespace
kubectl create namespace ilum

# Dependency namespaces
kubectl create namespace ilum-mongodb
kubectl create namespace ilum-kafka
kubectl create namespace ilum-minio
kubectl create namespace ilum-postgresql
kubectl create namespace ilum-kps
kubectl create namespace ilum-logs

Security keys

This application uses JSON Web Tokens (JWT) for authentication purposes. By default, the application employs an RSA key pair, which is randomly generated at runtime, to sign these tokens.

In its standard configuration, the application creates a fresh RSA key pair each time it starts. This approach simplifies local development and testing by automatically handling the key generation process. However, it must be emphasized that this approach is not suitable for a production environment.

The primary issue with using randomly generated keys in a production environment is the lack of persistence. Each time the application restarts, it generates a new RSA key pair, invalidating all previously issued tokens. This could lead to an abrupt and unanticipated logout for all users, disrupting user experience and potentially leading to data loss.

Generate private key

For a production environment, a stable and secure key pair should be manually generated and used consistently. This ensures that tokens remain valid across multiple application restarts, thus providing a consistent user experience.

You can generate an RSA key pair manually using tools like OpenSSL. A common command to generate a 2048-bit RSA private key is as follows:

openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -pkeyopt rsa_keygen_pubexp:65537 | \
  openssl pkcs8 -topk8 -nocrypt -outform pem > private-key.p8

The contents of the private key should look like the following:

In order to use private key as the setting ilum-core.security.jwt.privateKey, remove header and footer from the key.

Generate public key

To generate the corresponding public key, use:

openssl pkey -pubout -inform pem -outform pem -in private-key.p8 -out public-key.spki

The contents of the public key should look like the following:

In order to use public key as the setting ilum-core.security.jwt.publicKey, remove header and footer from the key.

Authentication Methods

Ilum as Identity Provider

Ilum can be deployed as an Identity Provider. With this feature, you can manage users exclusively within Ilum and authenticate them across other microservices such as Airflow, Superset, Grafana, Gitea, and Minio.

To enable Ilum's Identity Provider, add the following flags: --set global.security.hydra.enabled=true and --set global.security.hydra.uiUrl=<your-ilum-ui-domain>.

To learn more about Identity Provider configuration, visit this page

OpenLDAP

OpenLDAP is the open-source LDAP server. It is not enabled in Ilum by default. To enable it add --set openldap.enabled=true to your installation command.

If you want to access OpenLDAP you can use port-forward command kubectl port-forward svc/ilum-openldap 1389:389.

Ilum-Core is preconfigured to connect to OpenLDAP server deployed by Ilum. However you must switch security type to LDAP by adding --set ilum-core.security.type=ldap to your installation command.

To learn more about OpenLDAP and LDAP in Ilum, visit the LDAP documentation page here.

Dependency Deployment

Note: The components described below are required or recommended dependencies for Ilum. You can choose to deploy them using the provided instructions or integrate with your own existing managed services.

MongoDB

Deploy MongoDB in a dedicated namespace with high availability

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create MongoDB Values File

# mongodb-values.yaml
image:
  registry: docker.io
  repository: ilum/mongodb
  tag: 6.0.5
fullnameOverride: "ilum-mongodb"
architecture: "replicaset"
auth:
  enabled: false
volumePermissions:
  enabled: true

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions

helm install mongodb bitnami/mongodb \
  --namespace ilum-mongodb \
  --version 13.18.5 \
  --values mongodb-values.yaml

For more information about mongo deployment refer to mongodb chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled mongo
mongodb:
  enabled: false

ilum-core:
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
         # or use your mongo instance uri

Apache Kafka

Deploy Kafka cluster in a dedicated namespace

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create Kafka Values File

# kafka-values.yaml
extraConfig: |
  auto.create.topics.enable=false
  message.max.bytes=20000000
fullnameOverride: "ilum-kafka"
listeners:
  client:
    protocol: PLAINTEXT

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions

helm install kafka bitnami/kafka \
  --namespace ilum-kafka \
  --version 25.3.5 \
  --values kafka-values.yaml

For more information about kafka deployment refer to kafka chart details

4. Ilum AIO chart configuration values that need to be changed

ilum-core:
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
    # or use your kafka instance address

MinIO Deployment

Deploy MinIO cluster for object storage:

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create MinIO Values File

# minio-values.yaml
extraEnvVars: |
  - name: MINIO_BROWSER_REDIRECT_URL
    value: "http://ilum-minio.ilum-minio:9001/external/minio/"
 # Environment variables for hydra
  - name: MINIO_IDENTITY_OPENID_CONFIG_URL
    value: "ILUM_UI_PROTOCOL://ILUM_UI_DOMAIN/external/hydra/.well-known/openid-configuration"
  - name: MINIO_IDENTITY_OPENID_CLIENT_ID
    value: "HYDRA_CLIENT_ID"
  - name: MINIO_IDENTITY_OPENID_CLIENT_SECRET
    value: "HYDRA_CLIENT_SECRET"
  - name: MINIO_IDENTITY_OPENID_REDIRECT_URI
    value: "ILUM_UI_PROTOCOL://ILUM_UI_DOMAIN/external/minio/oauth_callback"
  - name: MINIO_IDENTITY_OPENID_SCOPES
    value: "openid,profile,email"
  - name:  MINIO_IDENTITY_OPENID_CLAIM_NAME
    value: "minio_policies"
 # end of hydra related environment variables
fullnameOverride: "ilum-minio"
defaultBuckets: "ilum-files, ilum-data, ilum-tables, ilum-mlflow, ilum-kestra"
auth:
  rootUser: "minioadmin"
  rootPassword: "minioadmin"
persistence:
  size: 16Gi

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions

helm install minio bitnami/minio \
  --namespace ilum-minio \
  --version 15.0.7 \
  --values minio-values.yaml

For more information about minio deployment refer to minio chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled minio
minio:
  enabled: false

ilum-core:
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

# If hive metastore is deployed with minio as the storage
ilum-hive-metastore:
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

# If trino is deployed with minio as the storage
trino:
  catalogs:
    ilum-delta: |
      connector.name=delta_lake
      delta.metastore.store-table-metadata=true
      delta.register-table-procedure.enabled=true
      hive.metastore.uri=thrift://ilum-hive-metastore:9083
      fs.native-s3.enabled=true
      s3.endpoint=http://ilum-minio.ilum-minio:9000
      s3.region=us-east-1
      s3.path-style-access=true
      s3.aws-access-key=minioadmin
      s3.aws-secret-key=minioadmin

# If Loki is deployed with s3 as the storage
loki:
  loki:
    auth_enabled: false
    storage:
      bucketNames:
        chunks: ilum-files
        ruler: ilum-files
        admin: ilum-files
      type: s3
      s3:
        endpoint: http://ilum-minio.ilum-minio:9000
        region: us-east-1
        secretAccessKey: minioadmin
        accessKeyId: minioadmin
        s3ForcePathStyle: true
        insecure: true

# If minio is deployed
minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

# If airflow is deployed with minio as the logs storage
airflow:
  extraEnv: |
    - name: AIRFLOW__API__AUTH_BACKENDS
      value: "airflow.api.auth.backend.default"
    - name: MINIO_USERNAME
      value: "minioadmin"
    - name: MINIO_PASSWORD
      value: "minioadmin"
    - name: MINIO_ENDPOINT
      value: "http://ilum-minio.ilum-minio:9000"

# If mlflow is deployed
mlflow:
  externalS3:
    host: "ilum-minio.ilum-minio"

# If kestra is deployed with minio as the storage
kestra:
  configuration:
    kestra:
      storage:
        type: minio
        minio:
          endpoint: ilum-minio.ilum-minio
          port: 9000
          secure: false
          accessKey: minioadmin
          secretKey: minioadmin
          region: "default"
          bucket: ilum-kestra

PostgreSQL Deployment

Deploy PostgreSQL for metadata storage:

1. Add Bitnami Repository

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

2. Create PostgreSQL Values File

# postgresql-values.yaml
fullnameOverride: ilum-postgresql
auth:
  postgresPassword: "CHANGEMEPLEASE"
  username: ilum
  password: "CHANGEMEPLEASE"

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions

helm install postgresql bitnami/postgresql \
  --namespace ilum-postgresql \
  --version 13.4.4 \
  --values postgresql-values.yaml

For more information about postgresql deployment refer to postgresql chart details

4. Ilum AIO chart configuration values that need to be changed

# Disable bundled PostgreSQL
postgresql:
  enabled: false

#Postgres extensions for databases auto-creation
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,airflow,metastore,mlflow,mlflow_auth,superset,gitea,n8n,hydra,kestra
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

# Configure hydra database if hydra is enabled in ilum-core deployment
ilum-core:
  hydra:
    dns: postgres://ilum:CHANGEMEPLEASE@ilum-postgresql.ilum-postgresql:5432/hydra?sslmode=disable

# If ilum-hive-metastore is deployed
ilum-hive-metastore:
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# If lineage is enabled
ilum-marquez:
  marquez:
    db:
      image: "bitnami/postgresql:16"
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"
      port: 5432
      name: marquez
      user: ilum
      password: "CHANGEMEPLEASE"

# If gitea is deployed
gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

# If airflow is deployed with postgres as metadata connection
airflow:
  data:
    metadataConnection:
      user: postgres
      pass: "CHANGEMEPLEASE"
      protocol: postgresql
      host: "ilum-postgresql-hl.ilum-postgresql"
      port: 5432
      db: airflow
      sslmode: disable

# If superset is deployed with postgres as the storage
superset:
  supersetNode:
    connections:
      db_host: "ilum-postgresql-hl.ilum-postgresql"
      db_port: "5432"
      db_user: ilum
      db_pass: "CHANGEMEPLEASE"
      db_name: superset

# If mlflow is deployed with postgres as the external database
mlflow:
  externalDatabase:
    dialectDriver: "postgresql"
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    user: ilum
    database: mlflow
    authDatabase: mlflow_auth
    existingSecret: "ilum-postgresql"
    existingSecretPasswordKey: "postgres-password"

# If n8n is deployed
n8n:
  db:
    type: postgresdb
  externalPostgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    username: "ilum"
    password: "CHANGEMEPLEASE"
    database: "n8n"

# If kestra is deployed
kestra:
  configuration:
    datasources:
      postgres:
        url: "jdbc:postgresql://ilum-postgresql-hl.ilum-postgresql:5432/kestra"
        driverClassName: "org.postgresql.Driver"
        username: ilum
        password: "CHANGEMEPLEASE"
  initContainers:
    - name: wait-for-postgres
      image: "bitnami/postgresql:16"
      command:
        - /bin/bash
      args:
        - -ec
        - |
          until pg_isready -h ilum-postgresql-hl.ilum-postgresql -p 5432 -U ilum -d kestra; do
            echo "Waiting for PostgreSQL server to be ready..."
            sleep 2
          done
          echo "Database is ready!"

Kube-Prometheus-Stack Deployment (If you plan to use it)

Deploy Kube-Prometheus-Stack for monitoring purposes:

1. Add Prometheus Community Repository

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

2. Create Kube-Prometheus-Stack Values File

# kps-values.yaml
nameOverride: "ilum-kps"
kubeStateMetrics:
  enabled: false
nodeExporter:
  enabled: false
alertmanager:
  enabled: false
grafana:
  nameOverride: "ilum-grafana"
  grafana.ini:
    server:
      root_url: "%(protocol)s://%(domain)s:%(http_port)s/external/grafana"
      serve_from_sub_path: "true"
    auth.generic_oauth:
      enabled: false
      name: Ilum
      allow_sign_up: true
      login_attribute_path: userId
      email_attribute_name: email
      role_attribute_path: contains(grafana_roles, 'Admin') && 'Admin' || contains(grafana_roles, 'Editor') && 'Editor' || contains(grafana_roles, 'Viewer') && 'Viewer'
      role_attribute_strict: true
  adminPassword: admin
  sidecar:
    dashboards:
      folderAnnotation: "grafana_folder"
      provider:
        foldersFromFilesStructure: "true"

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace ilum-kps \
  --version 54.2.2 \
  --values kps-values.yaml

For more information about kube-prometheus-stack deployment refer to kps chart details

4. Define prometheus PodMonitors and ServiceMonitors

You can define your own Pod and ServiceMonitors for prometheus. Example PodMonitors defined in Ilum AIO chart you can use

Ilum Pods Monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ilum-pods-monitor
  namespace: ilum-kps
  labels:
    ilum.prometheus: "true"
spec:
  selector:
    matchLabels:
      ilum.prometheus: "true"
  podMetricsEndpoints:
    - interval: 5s
      port: http
      path: /actuator/prometheus
  namespaceSelector:
    matchNames:
      - ilum

Spark Driver Pods Monitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: ilum-driver-pods-monitor
  namespace: ilum-kps
  labels:
    ilum.resourceType: driver
spec:
  selector:
    matchLabels:
      ilum.resourceType: driver
  podMetricsEndpoints:
    - interval: 5s
      port: spark-ui
      path: /metrics/executors/prometheus/
    - interval: 5s
      port: spark-ui
      path: /metrics/prometheus/
  namespaceSelector:
    any: true

Loki & Promtail Deployment (If you plan to use it)

Deploy Loki & Promtail for logs aggregation:

1. Add Grafana Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

2. Create Loki and Promtail Values File

# loki-values.yaml
nameOverride: ilum-loki
monitoring:
  selfMonitoring:
    enabled: false
    grafanaAgent:
      installOperator: false
    lokiCanary:
      enabled: false
test:
  enabled: false
loki:
  auth_enabled: false
  storage:
    bucketNames:
      chunks: ilum-files
      ruler: ilum-files
      admin: ilum-files
    type: s3
    s3:
      endpoint: http://ilum-minio.ilum-minio:9000
      region: us-east-1
      secretAccessKey: minioadmin
      accessKeyId: minioadmin
      s3ForcePathStyle: true
      insecure: true
  compactor:
    retention_enabled: true
    deletion_mode: filter-and-delete
    shared_store: s3
  limits_config:
    allow_deletes: true

# promtail-values.yaml
config:
  clients:
    - url: http://ilum-loki-write:3100/loki/api/v1/push
  snippets:
    pipelineStages:
      - match:
          selector: '{ilum_logAggregation!="true"}'
          action: drop
          drop_counter_reason: non_ilum_log
    extraRelabelConfigs:
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_ilum(.*)"
        replacement: "ilum${1}"
      - action: labelmap
        regex: "__meta_kubernetes_pod_label_spark(.*)"
        replacement: "spark${1}"

    scrapeConfigs: |
      - job_name: kubernetes-pods
        pipeline_stages:
          {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names:
              - ilum
        relabel_configs:
          - source_labels:
              - __meta_kubernetes_pod_controller_name
            regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
            action: replace
            target_label: __tmp_controller_name
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_name
              - __meta_kubernetes_pod_label_app
              - __tmp_controller_name
              - __meta_kubernetes_pod_name
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: app
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_instance
              - __meta_kubernetes_pod_label_instance
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: instance
          - source_labels:
              - __meta_kubernetes_pod_label_app_kubernetes_io_component
              - __meta_kubernetes_pod_label_component
            regex: ^;*([^;]+)(;.*)?$
            action: replace
            target_label: component
          {{- if .Values.config.snippets.addScrapeJobLabel }}
          - replacement: kubernetes-pods
            target_label: scrape_job
          {{- end }}
          {{- toYaml .Values.config.snippets.common | nindent 4 }}
          {{- with .Values.config.snippets.extraRelabelConfigs }}
          {{- toYaml . | nindent 4 }}
          {{- end }}

3. Deploy Loki, Ilum by default uses chart in 3.5.0 version but should also work with newer versions

helm install loki grafana/loki \
  --namespace ilum-logs \
  --version 3.5.0 \
  --values loki-values.yaml

For more information about loki deployment refer to loki chart details

4. Deploy Promtail, Ilum by default uses chart in ^6.15.x version but should also work with newer versions

helm install promtail grafana/promtail \
  --namespace ilum-logs \
  --version 6.15.5 \
  --values promtail-values.yaml

For more information about promtail deployment refer to promtail chart details

5. Ilum AIO chart configuration values that need to be changed

global:
  logAggregation:
    enabled: true
    loki:
      enabled: false
      url: http://ilum-loki-gateway.ilum-logs
    promtail:
      enabled: false

Pre-configured Stack Examples

Note: In those examples security configuration has been ignored, so those deployments stick to Ilum internal security mechanism, if you want to know how to use other security methods refer to LDAP or OAUTH2

Option 1: Minimal Production Stack

Suitable for deployments with basic requirements. Extra enabled modules:

ilum-sql
ilum-hive-metastore
ilum-marquez (lineage)

Minimal needed values file taking into account the components deployment described above. For more detailed configuration you can examine dedicated charts, for example ilum-core

# minimal-production-values.yaml
global:
  lineage:
    enabled: true

ilum-core:
  security:
    jwt:
      privateKey: Key generated as described in Security-keys section
      publicKey: Key generated as described in Security-keys section
  sql:
    enabled: true
  metastore:
    enabled: true
    type: hive
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

ilum-marquez:
  marquez:
    db:
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"

# Disable bundled dependencies
mongodb:
  enabled: false
kafka:
  enabled: false
minio:
  enabled: false
postgresql:
  enabled: false

# Above-default modules
ilum-sql:
  enabled: true

ilum-hive-metastore:
  enabled: true
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# Ilum add-ons
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,airflow,metastore,mlflow,mlflow_auth,superset,gitea,n8n,hydra,kestra
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

Option 2: Enterprise Production Stack

Comprehensive setup with monitoring, logging aggregation, and more optional modules. Extra enabled modules:

ilum-sql
ilum-hive-metastore
ilum-marquez (lineage)
superset
mlflow
kube-prometheus-stack
Loki & Promtail (log aggregation)

Minimal needed values file taking into account the components deployment described above. For more detailed configuration you can examine dedicated charts, for example ilum-core

# enterprise-production-values.yaml
global:
  lineage:
    enabled: true
  logAggregation:
    enabled: true
    loki:
      enabled: false
      url: http://ilum-loki-gateway.ilum-logs
    promtail:
      enabled: false

ilum-core:
  security:
    jwt:
      privateKey: Key generated as described in Security-keys section
      publicKey: Key generated as described in Security-keys section
  sql:
    enabled: true
  metastore:
    enabled: true
    type: hive
  mongo:
    uri: "mongodb://ilum-mongodb-0.ilum-mongodb-headless.ilum-mongodb:27017,ilum-mongodb-1.ilum-mongodb-headless.ilum-mongodb:27017/ilum?replicaSet=rs0"
  communication:
    type: kafka
  kafka:
    address: "ilum-kafka-controller-0.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-1.ilum-kafka-controller-headless.ilum-kafka:9092,ilum-kafka-controller-2.ilum-kafka-controller-headless.ilum-kafka:9092"
  minio:
    statusProbe:
      baseUrl: "http://ilum-minio.ilum-minio:9000"
  kubernetes:
    storage:
      type: s3
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"

gitea:
  initPreScript: "while ! nc -z ilum-postgresql-hl.ilum-postgresql 5432 2>/dev/null; do echo 'Waiting for Postgres...'; sleep 5; done; echo 'Postgres is ready!'"
  gitea:
    config:
      server:
        ROOT_URL: http://gitea.example.com/external/gitea
      database:
        DB_TYPE: postgres
        HOST: "ilum-postgresql-hl.ilum-postgresql:5432"
        NAME: gitea
        USER: ilum
        PASSWD: "CHANGEMEPLEASE"

ilum-marquez:
  marquez:
    db:
      host: "ilum-postgresql-0.ilum-postgresql-hl.ilum-postgresql"

# Disable bundled dependencies
mongodb:
  enabled: false
kafka:
  enabled: false
minio:
  enabled: false
postgresql:
  enabled: false

# Above-default modules
ilum-sql:
  enabled: true

ilum-hive-metastore:
  enabled: true
  storage:
    metastore:
      warehouse: "s3a://ilum-data/"
    type: "s3"
    s3:
      host: "ilum-minio.ilum-minio"
      port: 9000
      accessKey: "minioadmin"
      secretKey: "minioadmin"
  postgresql:
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    database: metastore
    auth:
      username: ilum
      password: "CHANGEMEPLEASE"

# Ilum add-ons
postgresExtensions:
  host: "ilum-postgresql-hl.ilum-postgresql"
  port: 5432
  databasesToCreate: marquez,metastore,mlflow,mlflow_auth,superset,gitea
  auth:
    username: ilum
    password: "CHANGEMEPLEASE"

minioExtensions:
  host: http://ilum-minio.ilum-minio:9000

superset:
  enabled: true
  supersetNode:
    connections:
      db_host: "ilum-postgresql-hl.ilum-postgresql"
      db_port: "5432"
      db_user: ilum
      db_pass: "CHANGEMEPLEASE"
      db_name: superset

mlflow:
  enabled: true
  externalS3:
    host: "ilum-minio.ilum-minio"
  externalDatabase:
    dialectDriver: "postgresql"
    host: "ilum-postgresql-hl.ilum-postgresql"
    port: 5432
    user: ilum
    database: mlflow
    authDatabase: mlflow_auth
    existingSecret: "ilum-postgresql"
    existingSecretPasswordKey: "postgres-password"

Helm Values Configuration

Complete Production Values Template

Reference the official Helm chart values at: https://artifacthub.io/packages/helm/ilum/ilum?modal=values

Ilum AIO Installation

Step 1: Verify Dependencies

# Check MongoDB
kubectl get pods -n ilum-mongodb
kubectl logs -f mongodb-0 -n ilum-mongodb

# Check Kafka
kubectl get pods -n ilum-kafka
kubectl logs -f kafka-0 -n ilum-kafka

# Check MinIO
kubectl get pods -n ilum-minio
kubectl logs -f minio-0 -n ilum-minio

# Check PostgreSQL
kubectl get pods -n ilum-postgresql
kubectl logs -f postgresql-primary-0 -n ilum-postgresql

Step 2: Deploy Ilum

Ilum in defragmented deployment needs ilum-minio secret

kubectl -n ilum create secret generic ilum-minio \
  --from-literal=root-user=minioadmin \
  --from-literal=root-password=minioadmin

ilum-postgresql secret

kubectl -n ilum create secret generic ilum-postgresql \
--from-literal=password=CHANGEMEPLEASE \
--from-literal=postgres-password=CHANGEMEPLEASE

Install Ilum

helm install ilum ilum/ilum \
  --namespace ilum \
  --values production-values.yaml

# Monitor deployment
kubectl get pods -n ilum -w

Post-Installation Configuration

1. Verify Installation

# Check all pods
kubectl get pods --all-namespaces | grep ilum

# Check services
kubectl get services --all-namespaces | grep ilum

# Check ingress (if enabled)
kubectl get ingress --all-namespaces

2. Access Ilum UI

# Port forward (for testing)
kubectl port-forward svc/ilum-ui 9777:9777 -n ilum

# Or access via ingress
http://ilum.company.com

# Or access via NodePort
kubectl get svc
http://K8S_NODE_ADDRESS:ILUM_UI_NODEPORT

Troubleshooting

Image Pulling Errors

During the installation of Ilum on your cluster, Helm will pull Docker images, which may be as large as 10 GB, depending on the additional modules you enable. Consequently, with a slow internet connection, you might encounter Image Pull Timeout errors if the image download time exceeds the configured timeout. To resolve this issue, you can:

Pull Docker image manually by running:

minikube ssh docker pull image
# for example
minikube ssh docker pull ilum/core-6.1.3

Change the image pull timeout in your kubernetes configurations like this:

minikube start --extra-config=kubelet.runtime-request-timeout=5m

or like this:

minikube start --extra-config=kubelet.image-pull-progress-deadline=5m

Default Passwords / Credentials

Ilum comes with predefined credentials for various modules to simplify initial setup and testing. However, for production deployments, it is critical to change these default credentials to ensure security and prevent unauthorized access.

Default Credentials

Application	Default Username	Default Password
Ilum UI	admin	admin
MinIO Console	minioadmin	minioadmin
Airflow Web UI	admin	admin
Superset UI	admin	admin
Gitea UI	ilum	ilum
Grafana	admin	admin

Database Credentials (For Internal Use)

Database	Default Username	Default Password
PostgreSQL	postgres	CHANGEMEPLEASE
Marquez	postgres	CHANGEMEPLEASE
Hive Metastore	postgres	CHANGEMEPLEASE

Table of Contents​

Overview​

Critical Components for Namespace Separation​

Kubernetes Prerequisites​

Supported Platforms​

Minimum Suggested Requirements​

Air-gapped (Offline) Environments​

Testing vs Production​

Architecture​

Components and modules​

MongoDB​

Apache Kafka​

MinIO​

PostgreSQL​

Ilum-Livy-proxy​

Jupyter​

Apache Zeppelin​

Hive Metastore​

Project Nessie​

Ilum SQL​

Trino​

n8n​

Apache Airflow​

Marquez​

Kestra​

Mage​

NiFi​

Streamlit​

Kube Prometheus Stack​

Loki and Promtail​

Production Architecture Overview​

Network Communication​

High Availability Configuration​

Failure Domains & Resilience​

Stateless Core Design​

Pod Anti-Affinity​

Namespace-Level Isolation​

Leader Election​

Dynamic Resource Scaling​

Spark Dynamic Allocation​

Kubernetes Horizontal Pod Autoscaler (HPA)​

Cluster Autoscaler Integration​

Auto-Pause on Idle​

Namespace Separation Strategy​

Recommended Namespace Structure​

Namespace Creation​

Security keys​

Generate private key​

Generate public key​

Authentication Methods​

Ilum as Identity Provider​

OpenLDAP​

Dependency Deployment​

MongoDB​

1. Add Bitnami Repository​

2. Create MongoDB Values File​

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

Apache Kafka​

1. Add Bitnami Repository​

2. Create Kafka Values File​

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

MinIO Deployment​

1. Add Bitnami Repository​

2. Create MinIO Values File​

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

PostgreSQL Deployment​

1. Add Bitnami Repository​

2. Create PostgreSQL Values File​

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions​

4. Ilum AIO chart configuration values that need to be changed​

Kube-Prometheus-Stack Deployment (If you plan to use it)​

1. Add Prometheus Community Repository​

2. Create Kube-Prometheus-Stack Values File​

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions​

4. Define prometheus PodMonitors and ServiceMonitors​

Loki & Promtail Deployment (If you plan to use it)​

1. Add Grafana Repository​

Table of Contents

Overview

Critical Components for Namespace Separation

Kubernetes Prerequisites

Supported Platforms

Minimum Suggested Requirements

Air-gapped (Offline) Environments

Testing vs Production

Architecture

Components and modules

MongoDB

Apache Kafka

MinIO

PostgreSQL

Ilum-Livy-proxy

Jupyter

Apache Zeppelin

Hive Metastore

Project Nessie

Ilum SQL

Trino

n8n

Apache Airflow

Marquez

Kestra

Mage

NiFi

Streamlit

Kube Prometheus Stack

Loki and Promtail

Production Architecture Overview

Network Communication

High Availability Configuration

Failure Domains & Resilience

Stateless Core Design

Pod Anti-Affinity

Namespace-Level Isolation

Leader Election

Dynamic Resource Scaling

Spark Dynamic Allocation

Kubernetes Horizontal Pod Autoscaler (HPA)

Cluster Autoscaler Integration

Auto-Pause on Idle

Namespace Separation Strategy

Recommended Namespace Structure

Namespace Creation

Security keys

Generate private key

Generate public key

Authentication Methods

Ilum as Identity Provider

OpenLDAP

Dependency Deployment

MongoDB

1. Add Bitnami Repository

2. Create MongoDB Values File

3. Deploy MongoDB, Ilum by default uses chart in ^13.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

Apache Kafka

1. Add Bitnami Repository

2. Create Kafka Values File

3. Deploy Kafka, Ilum by default uses chart in ^25.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

MinIO Deployment

1. Add Bitnami Repository

2. Create MinIO Values File

3. Deploy MinIO, Ilum by default uses chart in ^15.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

PostgreSQL Deployment

1. Add Bitnami Repository

2. Create PostgreSQL Values File

3. Deploy PostgreSQL, Ilum by default uses chart in ^13.x version but should also work with newer versions

4. Ilum AIO chart configuration values that need to be changed

Kube-Prometheus-Stack Deployment (If you plan to use it)

1. Add Prometheus Community Repository

2. Create Kube-Prometheus-Stack Values File

3. Deploy kube-prometheus-stack, Ilum by default uses chart in ^54.x version but should also work with newer versions

4. Define prometheus PodMonitors and ServiceMonitors

Loki & Promtail Deployment (If you plan to use it)

1. Add Grafana Repository