Docs on Krkn

Krkn

Thu, 05 Jan 2017 00:00:00 +0000

Installation

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to krkn in anyway in the future we recommend forking the repository first before cloning.

See the latest release version here

$ git clone https://github.com/krkn-chaos/krkn.git --branch <release version>
$ cd krkn

Fork and Clone the Repository

Fork the repository

Krkn-lib

Thu, 05 Jan 2017 00:00:00 +0000

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

The Library contains Classes, Models and helper functions used in Kraken to interact with Kubernetes, Openshift and other external APIS. The goal of this library is to give to developers the building blocks to realize new Chaos Scenarios and to increase the testability and the modularity of the Krkn codebase.

Adding scenarios via plugin api

Mon, 01 Jan 0001 00:00:00 +0000

Scenario Plugin API:

This API enables seamless integration of Scenario Plugins for Krkn. Plugins are automatically detected and loaded by the plugin loader, provided they extend the AbstractPluginScenario abstract class, implement the required methods, and adhere to the specified naming conventions.

Plugin folder:

The plugin loader automatically loads plugins found in the krkn/scenario_plugins directory, relative to the Krkn root folder. Each plugin must reside in its own directory and can consist of one or more Python files. The entry point for each plugin is a Python class that extends the AbstractPluginScenario abstract class and implements its required methods.

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started with Krkn-AI

This documentation details how to deploy a sample microservice application on Kubernetes Cluster and run Krkn-AI test.

Prerequisites

Follow this guide to install Krkn-AI CLI.
Krkn-AI uses Thanos Querier to fetch SLO metrics by PromQL. You can easily install it by setting up prometheus-operator in your cluster.

Deploy Sample Microservice

For demonstration purpose, we will deploy a sample microservice called robot-shop on the cluster:

krkn-hub

Thu, 05 Jan 2017 00:00:00 +0000

Hosts container images and wrapper for running scenarios supported by Krkn, a chaos testing tool for Kubernetes clusters to ensure it is resilient to failures. All we need to do is run the containers with the respective environment variables defined as supported by the scenarios without having to maintain and tweak files!

Set Up

You can use docker or podman to run kraken-hub

Adding New Scenario to Krkn-hub

Mon, 01 Jan 0001 00:00:00 +0000

Adding/Editing a New Scenario to Krkn-hub

Create folder with scenario name under krkn-hub
Create generic scenario template with environment variables

a. See scenario.yaml for example

b. Almost all parameters should be set using a variable (these will be set in the env.sh file or through the command line environment variables)
Add defaults for any environment variables in an “env.sh” file

a. See env.sh for example
Create script to run.sh chaos scenario a. See run.sh for example

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

TL;DR

# 1. Install krknctl
curl -fsSL https://raw.githubusercontent.com/krkn-chaos/krknctl/refs/heads/main/install.sh | bash

# 2. Create a test workload
kubectl create namespace chaos-test
kubectl create deployment nginx-test --image=nginx --replicas=3 -n chaos-test

# 3. Run your first chaos scenario (pod disruption)
krknctl run pod-scenarios --namespace chaos-test --pod-label "app=nginx-test" --disruption-count 1

# 4. Verify pods recovered
kubectl get pods -n chaos-test -l app=nginx-test

What you need

Requirement	Minimum Version	Check Command
Kubernetes or OpenShift cluster	1.21+	`kubectl version`
kubeconfig with cluster-admin access	—	`kubectl get nodes`
Docker or Podman	Docker 20.10+ / Podman 4.0+	`docker --version` or `podman --version`

Basic Run

This is the best starting point if you are new to Krkn or want to explore a specific scenario quickly. No metrics, no scoring, no pipeline — just run a scenario and see what happens.

Metrics Validation

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Run chaos and automatically evaluate Prometheus metrics — getting a clear pass or fail without manual inspection.

This journey is well suited to CI/CD pipelines where you cannot watch the cluster in real time.

What you need

Everything from Basic Run
A Prometheus instance accessible from where Krkn runs (auto-detected on OpenShift; set via scenario flags on Kubernetes) — need to set one up? See installing Prometheus on a kind cluster
krknctl installed

Steps

Install krknctl — follow the installation guide.

Running a Chaos Scenario with Krkn

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios

Config

Instructions on how to setup the config and all the available options supported can be found at Config.

In all the examples below you’ll replace the scenario_type with the scenario plugin type that can be found in the second column here

Running a Single Scenario

To run a single scenario, you’ll edit the krkn config file and only have 1 item in the list of chaos_scenarios

Setting Up Disconnected Environment

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios in a Disconnected Environment

Mirror following images on the bastion host

quay.io/krkn-chaos/krkn-hub:node-scenarios-bm - Master/worker node disruptions on baremetal
quay.io/krkn-chaos/krkn-hub:network-chaos - Network disruptions/traffic shaping
quay.io/krkn-chaos/krkn-hub:pod-scenarios - Pod level disruptions and evaluating recovery time/availability
quay.io/krkn-chaos/krkn-hub:syn-flood - Generates substantial amount of traffic/half open connections targeting a service
quay.io/krkn-chaos/krkn-hub:node-cpu-hog - Hogs CPU on the target nodes
quay.io/krkn-chaos/krkn-hub:node-io-hog - Hogs IO on the target nodes
quay.io/krkn-chaos/krkn-hub:node-memory-hog - Hogs memory on the target nodes
quay.io/krkn-chaos/krkn-hub:pvc-scenarios - Fills up a given PersistentVolumeClaim
quay.io/krkn-chaos/krkn-hub:service-hijacking - Simulates fake HTTP response for a service
quay.io/krkn-chaos/krkn-hub:power-outages - Shuts down the cluster and turns it back on
quay.io/krkn-chaos/krkn-hub:container-scenarios - Kills a container via provided kill signal
quay.io/krkn-chaos/krkn-hub:application-outages - Isolates application Ingress/Egress traffic
quay.io/krkn-chaos/krkn-hub:time-scenarios - Tweaks time/date on the nodes
quay.io/krkn-chaos/krkn-hub:pod-network-chaos - Introduces network chaos at pod level
quay.io/krkn-chaos/krkn-hub:node-network-filter - Node ip traffic filtering
quay.io/krkn-chaos/krkn-hub:pod-network-filter - DNS, internal/external service outages

Will also need these mirrored images inside the cluster

Network disruptions - quay.io/krkn-chaos/krkn:tools
Hog scenarios ( CPU, Memory and IO ) - quay.io/krkn-chaos/krkn-hog
SYN flood - quay.io/krkn-chaos/krkn-syn-flood:latest
Pod network filter scenarios - quay.io/krkn-chaos/krkn-network-chaos:latest
Service hijacking scenarios - quay.io/krkn-chaos/krkn-service-hijacking:v0.1.3

How to Mirror

The strategy is simple:

Adding New Scenario to Krknctl

Mon, 01 Jan 0001 00:00:00 +0000

Adding a New Scenario to Krknctl

For krknctl to find the parameters of the scenario it uses a krknctl input json file. Once this file is added to krkn-hub, krknctl will be able to find it along with the details of how to run the scenario.

Add KrknCtl Input Json

This file adds every environment variable that is set up for krkn-hub to be defined as a flag to the krknctl cli command. There are a number of different type of variables that you can use, each with their own required fields. See below for an example of the different variable types

Git Help For Contributions

Mon, 01 Jan 0001 00:00:00 +0000

How to contribute

Contributions are always appreciated.

How to:

Pull request

In order to submit a change or a PR, please fork the project and follow instructions:

$ git clone http://github.com/<me>/krkn-hub
$ cd krkn-hub
$ git checkout -b <branch_name>
$ <make change>
$ git add <changes>
$ git commit -a
$ <insert good message>
$ git push

Squash Commits

If there are multiple commits, please rebase/squash multiple commits before creating the PR by following:

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Choose Your Installation Method

Krkn provides multiple ways to run chaos scenarios. Choose the method that best fits your needs:

Tool	What is it?	Best For
krknctl	CLI tool with auto-completion	Complex workflow orchestration, querying and running scenarios, ease of use
krkn-hub	Pre-built container images	CI/CD pipelines, automation
krkn Dashboard	Web UI for running scenarios	Visual runs, demos, teams that prefer a GUI
krkn	Standalone Python program	Full control, development, and customization

Recommendation

krknctl is the recommended way to run Krkn. It provides the simplest path to chaos testing with powerful capabilities including complex workflow orchestration, built-in scenario discovery, and interactive query support — all without managing configuration files.

Recommendation

Look for features marked with [BETA] (e.g., [BETA] Krkn Resilience Score). Beta features provide early access to new capabilities for experimentation and feedback and may not yet meet the stability, performance, or compatibility guarantees of GA features. Please refer to the Beta feature policy for more details.

Installation Methods

krknctl (Recommended)

What is it? A dedicated command-line interface (CLI) tool that simplifies running Krkn chaos scenarios while providing powerful orchestration capabilities.

Long-Term Storage

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Persist metrics from every chaos run into Elasticsearch so you can compare behavior across releases, dates, or cluster configurations.

This journey enables regression analysis — for example, detecting that API server latency during a node failure has increased between software versions.

What you need

Everything from Metrics Validation
An Elasticsearch instance (self-hosted or managed) — need to set one up? See installing Elasticsearch on a kind cluster
krknctl installed

Steps

Complete Metrics Validation first to confirm Prometheus evaluation is working.

Adding to Krkn Test Suite

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers how to add both unit tests and functional tests to the krkn project. Tests are essential for ensuring code quality and preventing regressions.

Unit Tests

Unit tests in krkn are located in the tests/ directory and use Python’s unittest framework with comprehensive mocking to avoid requiring external dependencies like cloud providers or Kubernetes clusters.

Creating a Unit Test

1. File Location and Naming

Place your test file in the tests/ directory with the naming pattern test_<feature>.py:

Krkn-AI

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Prerequisites

Python 3.11+
Podman or Docker Container Runtime
krknctl
uv package manager (recommended) or pip

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to Krkn-AI in anyway in the future we recommend forking the repository first before cloning.

$ git clone https://github.com/krkn-chaos/krkn-ai.git
$ cd krkn-ai

Fork and Clone the Repository

Fork the repository

License Header & Pre-Commit

Mon, 01 Jan 0001 00:00:00 +0000

License Header & Pre-Commit

Krkn enforces Apache 2.0 license headers on all Python files using a pre-commit hook. Contributors must have this set up before submitting pull requests.

Setup (one-time)

pip install pre-commit
pre-commit install

Check license headers manually

python scripts/check_license.py

Auto-fix all missing headers

python scripts/check_license.py --fix

Run via pre-commit against all files

pre-commit run check-license-header --all-files

Resilience Score

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Generate a numerical score (0–100%) that represents how well your environment held up during chaos — giving you more signal than a binary pass/fail.

A resilience score lets you track improvement over time, compare environments, and set score thresholds as release gates.

Beta Feature

Resiliency Scoring is currently in Beta. The configuration format and scoring behavior may change in future releases.

What you need

Everything from Metrics Validation
krknctl installed

How scoring works

After a chaos scenario completes, Krkn evaluates a set of SLOs (defined as PromQL expressions) over the chaos time window. Each SLO is weighted by severity:

Testing your changes

Mon, 01 Jan 0001 00:00:00 +0000

This page gives details about how you can get a kind cluster configured to be able to run on krkn-lib (the lowest level of krkn-chaos repos) up through krknctl (our easiest way to run and highest level repo)

Configure Kind Testing Environment

Install kind
Create cluster using kind-config.yml under krkn-lib base folder

Krkn Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

The Krkn Dashboard is a web UI for running and observing Krkn chaos scenarios. You can run it locally (Node.js on your machine), containerized (Podman/Docker), or via krknctl.

Prerequisites (all methods)

Kubernetes cluster — You need a cluster and a kubeconfig so that the dashboard can target it for chaos runs. If you don’t have one, see Kubernetes, minikube, K3s, or OpenShift.
Podman or Docker — The dashboard starts chaos runs by launching krkn-hub containers; the host must have Podman (or Docker) installed and available.

Local installation

Run the dashboard on your machine with Node.js.

Multi-Cluster Orchestration

Mon, 01 Jan 0001 00:00:00 +0000

Goal: Run chaos scenarios across multiple clusters or cloud environments from a single control point — useful for validating a multi-region application or comparing cluster configurations.

The recommended approach for multi-cluster orchestration is krkn-operator: a Kubernetes operator that runs on a dedicated control plane cluster and dispatches chaos scenarios to any number of registered target clusters, without distributing credentials to individual users.

How krkn-operator works

A control plane cluster runs the operator and its web console
Target clusters are registered once by an administrator (via kubeconfig, service account token, or username/password)
Users select one or more targets through the web UI and launch scenarios — they never handle cluster credentials directly
Scenarios run in parallel across all selected targets

This design preserves the original Krkn architecture (chaos runs from outside the cluster) while adding a secure, centralized orchestration layer.

Cerberus

Mon, 01 Jan 0001 00:00:00 +0000

Cerberus

Guardian of Kubernetes and OpenShift Clusters

Cerberus watches the Kubernetes/OpenShift clusters for dead nodes, system component failures/health and exposes a go or no-go signal which can be consumed by other workload generators or applications in the cluster and act accordingly.

Workflow

Installation

Instructions on how to setup, configure and run Cerberus can be found at Installation.

Developers Guide

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to develop and add to Krkn. Before you start, it is recommended that you read the following documents first:

Be sure to properly install Krkn. Then you can start to develop krkn. The following documents will help you get started:

Add k8s functionality to krkn-lib
Add a New Chaos Scenario using Plugin API: Adding a new scenario into krkn
Test your changes

NOTE: All base kubernetes functionality should be added into krkn-lib and called from krkn

Docs on Krkn

Krkn

Installation

Clone the Repository

Fork and Clone the Repository

Krkn-lib

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

Contents

Adding scenarios via plugin api

Scenario Plugin API:

Plugin folder:

Getting Started

Getting Started with Krkn-AI

Prerequisites

Deploy Sample Microservice

krkn-hub

Set Up

Adding New Scenario to Krkn-hub

Adding/Editing a New Scenario to Krkn-hub

Getting Started

TL;DR

What you need

Basic Run

Metrics Validation

What you need

Steps

Running a Chaos Scenario with Krkn

Getting Started Running Chaos Scenarios

Config

Running a Single Scenario

Setting Up Disconnected Environment

Getting Started Running Chaos Scenarios in a Disconnected Environment

Mirror following images on the bastion host

Will also need these mirrored images inside the cluster

How to Mirror

Adding New Scenario to Krknctl

Adding a New Scenario to Krknctl

Add KrknCtl Input Json

Git Help For Contributions

How to contribute

Pull request

Squash Commits

Installation

Choose Your Installation Method

Recommendation

Recommendation

Installation Methods

krknctl (Recommended)

Long-Term Storage

What you need

Steps

Adding to Krkn Test Suite

Unit Tests

Creating a Unit Test

1. File Location and Naming

Krkn-AI

Installation

Prerequisites

Clone the Repository

Fork and Clone the Repository

License Header & Pre-Commit

License Header & Pre-Commit

Setup (one-time)

Check license headers manually

Auto-fix all missing headers

Run via pre-commit against all files

Resilience Score

Beta Feature

What you need

How scoring works

Testing your changes

Configure Kind Testing Environment

Krkn Dashboard

Prerequisites (all methods)

Local installation

Multi-Cluster Orchestration

How krkn-operator works

Cerberus

Cerberus