About the reviewers
Onur Yilmaz is a senior software engineer at a multinational enterprise software company. He is a Certified Kubernetes Administrator (CKA) and works on Kubernetes and cloud management systems as a keen supporter of cutting-edge technologies. Furthermore, he is the author of multiple books on Kubernetes, Docker, serverless architectures, and cloud-native continuous integration and delivery. In addition, he has one master’s degree and two bachelors’ degrees in the engineering field.
Sergei Bulavintsev is a senior DevOps engineer at GlobalDots. He is passionate about open source, cloud-native infrastructure, and tools that increase developers’ productivity. He has successfully migrated multiple customers to the cloud and Kubernetes, advocating for and implementing the GitOps approach. Sergei is also an active member of his local cloud-native community and holds industry certifications such as CKA, CKAD, CKS, and RHCA level 2.
I would like to thank my wife, Elena, and our two children, Maria and Maxim, for their support and patience. I would also like to thank my family, friends, and colleagues who have helped me become who I am today.
Sidecar pattern • 10
Ambassador pattern • 10
Adapter pattern • 11
Multi-node patterns • 11
Level-triggered infrastructure and reconciliation • 11
The Kubernetes APIs • 11
Resource categories • 12
Kubernetes components • 15
Control plane components • 15
Node components • 18
Kubernetes container runtimes
The Container Runtime Interface (CRI) • 18
Docker • 20 containerd • 22
CRI-O • 22
Lightweight VMs • 22
Summary
Installing Rancher Desktop • 26
Installation on macOS • 26
Installation on Windows • 26
Additional installation methods • 26
Meet kubectl • 27
Kubectl alternatives – K9S, KUI, and Lens • 28
K9S • 28
KUI • 29
• 29
Quick introduction to Minikube • 31
Installing Minikube • 31
Installing Minikube on Windows • 31
Installing Minikube on macOS • 35
Troubleshooting the Minikube installation • 37
Checking out the cluster • 38
Doing work • 40
Examining the cluster with the dashboard • 42
Creating a multi-node cluster with KinD
Quick introduction to KinD • 44
Installing KinD • 44
Dealing with Docker contexts • 44
Creating a cluster with KinD • 45
Doing work with KinD • 48
Accessing Kubernetes services locally through a proxy • 49
with k3d
Quick introduction to k3s and k3d • 50
Installing k3d • 50
Creating the cluster with k3d • 51
Minikube, KinD, and k3d
Honorable mention – Rancher Desktop Kubernetes cluster • 54
Creating clusters in the cloud (GCP, AWS, Azure, and Digital
The cloud-provider interface • 56
Creating Kubernetes clusters in the cloud • 56
GCP • 57
GKE Autopilot • 57
AWS • 57
Kubernetes on EC2 • 57
Amazon EKS • 58
Fargate • 59
Azure • 59
Digital Ocean • 59
Other cloud providers • 60
Once upon a time in China • 60
IBM Kubernetes service • 60
Oracle Container Service • 61
Creating a bare-metal
Use cases for bare metal • 61
When should you consider creating a bare-metal cluster? • 62
Understanding the process • 62
Using the Cluster API for managing bare-metal clusters • 62
Using virtual private cloud infrastructure • 63
Building your own cluster with Kubespray • 63
Building your cluster with Rancher RKE • 63
Running managed Kubernetes on bare metal or VMs • 63
GKE Anthos • 64
EKS Anywhere • 64
AKS Arc • 64
Redundancy • 68
Hot swapping • 68
Leader election • 68
Smart load balancing • 69
Idempotency • 69
Self-healing • 69
High availability best practices �����������������������������������������������������������������������������������������������
Creating highly available clusters • 70
Making your nodes reliable • 71
Protecting your cluster state • 72
Clustering etcd • 72
Protecting your data • 72
Running redundant API servers • 73
Running leader election with Kubernetes • 73
Making your staging environment highly available • 74
Testing high availability • 74
High availability, scalability, and
Installing the cluster autoscaler • 77
Considering the vertical pod autoscaler • 79
Autoscaling based on custom metrics • 80
Large cluster performance, cost, and design trade-offs
Best effort • 81
Maintenance windows • 81
Quick recovery • 82
Zero downtime • 82
Site reliability engineering • 84
Performance and data consistency • 84
Choosing and managing the cluster capacity
Choosing your node types • 85
Choosing your storage solutions • 85
Trading off cost and response time • 86
Using multiple node configurations effectively • 86
Benefiting from elastic cloud resources • 87
Autoscaling instances • 87
Mind your cloud quotas • 87
Manage regions carefully • 87
Considering container-native solutions • 88
Pushing the envelope with Kubernetes
Improving the performance and scalability of Kubernetes • 90
Caching reads in the API server • 90
The pod lifecycle event generator • 91
Serializing API objects with protocol buffers • 92
etcd3 • 92
gRPC instead of REST • 92
Leases instead of TTLs • 92
Watch implementation • 92
State storage • 92
Other optimizations • 93
Measuring the performance and scalability of Kubernetes • 93
The Kubernetes SLOs • 93
Measuring API responsiveness • 93
Measuring end-to-end pod startup time • 95
Testing Kubernetes at scale
Introducing the Kubemark tool • 97
Setting up a Kubemark cluster • 97
Comparing a Kubemark cluster to a real-world cluster • 98
Summary
Node challenges • 100
Network challenges • 101
Image challenges • 103
Configuration and deployment challenges • 104
Pod and container challenges • 105
Organizational, cultural, and process challenges • 105
Understanding service accounts in Kubernetes • 107
How does Kubernetes manage service accounts? • 108
Accessing the API server • 108
Authenticating users • 109
Impersonation • 111
Authorizing requests • 112
Using admission control plugins • 114
Securing pods • 115
Using a private image repository • 115
ImagePullSecrets • 115
Specifying a security context for pods and containers • 116
Pod security standards • 117
Protecting your cluster with AppArmor • 117
Writing AppArmor profiles • 119
Pod Security Admission • 121
Managing network policies • 121
Choosing a supported networking solution • 122
Defining a network policy • 122
Limiting egress to external networks • 124
Cross-namespace policies • 124
The costs of network policies • 124
Using secrets • 125
Storing secrets in Kubernetes • 125
Configuring encryption at rest • 125
Creating secrets • 126
Decoding secrets • 126
Using secrets in a container • 127
Managing secrets with Vault • 128
Running a multi-tenant cluster
The case for multi-tenant clusters • 129
Using namespaces for safe multi-tenancy • 129
Avoiding namespace pitfalls • 130
Using virtual clusters for strong multi-tenancy • 131
Summary
Chapter 5: Using Kubernetes Resources in Practice
Designing the Hue platform
Defining the scope of Hue • 138
Smart reminders and notifications • 138
Security, identity, and privacy • 138
Hue components • 139
User profile • 139
User graph • 139
Identity • 139
Authorizer • 140
External services • 140
Generic sensor • 140
Generic actuator • 140
User learner • 140
Hue microservices • 140
Plugins • 141
Data stores • 141
Stateless microservices • 141
Serverless functions • 141
Event-driven interactions • 141
Planning workflows • 142
Automatic workflows • 142
Human workflows • 142
Budget-aware workflows • 142
Using Kubernetes to build the Hue platform
Using kubectl effectively • 143
Understanding kubectl manifest files • 144
apiVersion • 144
kind • 144
metadata • 145
spec • 145
Deploying long-running microservices in pods • 146
Creating pods • 146
Decorating pods with labels • 147
Deploying long-running processes with deployments • 148
Updating a deployment • 151
Separating internal and external services
Deploying an internal service • 152
Creating the hue-reminders service • 154
Exposing a service externally • 155
Ingress • 156
Advanced scheduling
Node selector • 158
Taints and tolerations • 159
Node affinity and anti-affinity • 160
Pod affinity and anti-affinity • 161
Pod topology spread constraints • 162
The descheduler • 163
Using namespaces to limit access
Using Kustomization for hierarchical cluster
Understanding the basics of Kustomize • 166
Configuring the directory structure • 167
Applying Kustomizations • 168
Patching • 169
Kustomizing the entire staging namespace • 170
Launching jobs
Running jobs in parallel • 172
Cleaning up completed jobs • 173
Scheduling cron jobs • 174
Mixing non-cluster components
Outside-the-cluster-network components • 176
Inside-the-cluster-network components • 176
Managing the Hue platform with Kubernetes • 176
Using liveness probes to ensure your containers are alive • 177
Using readiness probes to manage dependencies • 178
Using startup probes • 178
Employing init containers for orderly pod bring-up • 179
Pod readiness and readiness gates • 180
Sharing with DaemonSet pods • 180
Evolving the Hue platform with Kubernetes
Utilizing Hue in an enterprise • 182
Advancing science with Hue • 182
Educating the kids of the future with Hue • 182
Summary
Chapter 6: Managing Storage
Persistent volumes walk-through
Understanding volumes • 186
Using emptyDir for intra-pod communication • 186
Using HostPath for intra-node communication • 188
Using local volumes for durable node storage • 190
Provisioning persistent volumes • 191
Creating persistent volumes • 192
Capacity • 193
Volume mode • 193
Access modes • 193
Reclaim policy • 194
Storage class • 194
Volume type • 194
Mount options • 194
Projected volumes • 195
serviceAccountToken projected volumes • 196
Creating a local volume • 196
Making persistent volume claims • 197
Mounting claims as volumes • 199
Raw block volumes • 200
CSI ephemeral volumes • 202
Generic ephemeral volumes • 202
Storage classes • 204
Default storage class • 205
Demonstrating persistent volume storage end to end
Public cloud storage volume types – GCE, AWS, and Azure
AWS Elastic Block Store (EBS) • 210
AWS Elastic File System (EFS) • 212
GCE persistent disk • 215
Google Cloud Filestore • 216
Azure data disk • 217
Azure file • 218
GlusterFS and Ceph volumes in Kubernetes
Using GlusterFS • 219
Creating endpoints • 220
Adding a GlusterFS Kubernetes service • 220
Creating pods • 220
Using Ceph • 221
Connecting to Ceph using RBD • 221
Rook • 223
Integrating enterprise storage into Kubernetes
Other storage providers • 228
The Container Storage Interface
Advanced storage features • 229
Volume snapshots • 229
CSI volume cloning • 230
Storage capacity tracking • 231
Volume health monitoring • 231
Summary
Stateful versus stateless applications in Kubernetes
Understanding the nature of distributed data-intensive apps • 236
Why manage the state in Kubernetes? • 236
Why manage the state outside of Kubernetes? • 236
Shared environment variables versus DNS records for discovery • 236
Accessing external data stores via DNS • 237
Accessing external data stores via environment variables • 237
Consuming a ConfigMap as an environment variable • 238
Using a redundant in-memory state • 238
Using DaemonSet for redundant persistent storage • 239
Applying persistent volume claims • 239
Utilizing StatefulSet • 239
Working with StatefulSets • 241
Running a Cassandra cluster in Kubernetes
A quick introduction to Cassandra • 243
The Cassandra Docker image • 244
Exploring the build.sh script • 246
Exploring the run.sh script • 247
Hooking up Kubernetes and Cassandra • 252
Digging into the Cassandra configuration file • 252
The custom seed provider • 253
Creating a Cassandra headless service • 254
Using StatefulSet to create the Cassandra cluster • 255
Dissecting the StatefulSet YAML file • 255
Chapter 8: Deploying and Updating Applications
Live cluster updates
Rolling updates • 262
Complex deployments • 264
Blue-green deployments • 265
Canary deployments • 266
Managing data-contract changes • 267
Migrating data • 267
Deprecating APIs • 267
Horizontal pod autoscaling
Creating a horizontal pod autoscaler • 269
Custom metrics • 272
Keda • 273
Autoscaling with kubectl • 274
Performing rolling updates with autoscaling • 276
Handling scarce resources with limits and quotas
Enabling resource quotas • 279
Resource quota types • 279
Compute resource quota • 280
Storage resource quota • 280
Object count quota • 281
Quota scopes • 282
Resource quotas and priority classes • 282
Requests and limits • 283
Working with quotas • 283
Using namespace-specific context • 283
Creating quotas • 283
Using limit ranges for default compute quotas • 287
Continuous integration and deployment
What is a CI/CD pipeline? • 289
Designing a CI/CD pipeline for Kubernetes • 290
Provisioning infrastructure for your applications
Cloud provider APIs and tooling • 291
Terraform • 291
Pulumi • 292
Custom operators • 294
Using Crossplane • 294
Summary
Chapter 9: Packaging Applications
The motivation for Helm • 298
The Helm 3 architecture • 298
Helm release secrets • 298
The Helm client • 299
The Helm library • 299
Helm 2 vs Helm 3 • 299
Using Helm
Installing Helm • 300
Installing the Helm client • 300
Finding charts • 300
Adding repositories • 301
Installing packages • 304
Checking the installation status • 305
Customizing a chart • 307
Additional installation options • 308
Upgrading and rolling back a release • 308
Deleting a release • 310
Working with repositories • 311
Managing charts with Helm • 311
Taking advantage of starter packs • 312
Creating your own charts
The Chart.yaml file • 313
Versioning charts • 313
The appVersion field • 313
Deprecating charts • 314
Chart metadata files • 314
Managing chart dependencies • 315
Utilizing additional subfields of the dependencies field • 316
Using templates and values • 317
Writing template files • 318
Testing and troubleshooting your charts • 320
Embedding built-in objects • 322
Feeding values from a file • 322
Kustomize • 324
Cue • 324
kapp-controller • 324
Intra-pod communication (container to container) • 326
Inter-pod communication (pod to pod) • 326
Pod-to-service communication • 326
Lookup and discovery • 328
Self-registration • 328
Services and endpoints • 328
Loosely coupled connectivity with queues • 328
Loosely coupled connectivity with data stores • 329
Kubernetes ingress • 329
DNS in Kubernetes • 330
CoreDNS • 332
Kubernetes network plugins
Basic Linux networking • 334
IP addresses and ports • 334
Network namespaces • 334
Subnets, netmasks, and CIDRs • 334
Virtual Ethernet devices • 334
Bridges • 334
Routing • 334
Maximum transmission unit • 335
Pod networking • 335
Kubenet • 335
Requirements • 335
Setting the MTU • 336
The CNI • 336
The container runtime • 337
The CNI plugin • 337
Kubernetes and eBPF
Kubernetes
Bridging on bare-metal clusters
The Calico project • 342
Weave Net • 342
Cilium • 342
341
Efficient IP allocation and routing • 342
Identity-based service-to-service communication • 343
Load balancing • 343
Bandwidth management • 343
Observability • 343 Using network
Understanding the Kubernetes network policy design
Network policies and CNI plugins • 344
Configuring network policies • 344
Implementing network policies • 345
External load balancers
347
Configuring an external load balancer • 347
Finding the load balancer IP addresses • 348
Preserving client IP addresses • 349
Understanding even external load balancing • 349
Service load balancers • 350
Ingress • 350
HAProxy • 352
MetalLB • 354
Traefik • 354
Kubernetes Gateway API • 355
Gateway API resources • 355
Attaching routes to gateways • 356
344
Gateway API in action
First look at the loopback plugin • 358
Building on the CNI plugin skeleton • 362
Reviewing the bridge plugin • 364
Chapter 11: Running Kubernetes on Multiple Clusters
Understanding stretched Kubernetes clusters •
Pros of a stretched cluster • 372
Cons of a stretched cluster • 372
Understanding multi-cluster Kubernetes • 372
Pros of multi-cluster Kubernetes • 373
Cons of multi-cluster Kubernetes • 373
The history of cluster federation in Kubernetes
Cluster API architecture • 374
Management cluster • 375
Work cluster • 375
Bootstrap provider • 376
Infrastructure provider • 376
Control plane • 376
Custom resources • 376
Karmada
Karmada architecture • 378
Karmada concepts • 378
ResourceTemplate • 379
PropagationPolicy • 379
OverridePolicy • 379
Additional capabilities • 379 Clusternet
Clusternet architecture • 380
Clusternet hub • 381
Clusternet scheduler • 381
Clusternet agent • 381
Multi-cluster deployment • 381
Clusterpedia architecture • 382
Clusterpedia API server • 383
ClusterSynchro manager • 383
Storage layer • 384
Storage component • 384
Importing clusters • 384
Advanced multi-cluster search • 384
Resource collections • 385
Open Cluster Management
OCM architecture • 386
OCM cluster lifecycle • 387
OCM application lifecycle • 387
OCM governance, risk, and compliance • 388
Virtual Kubelet
Tensile-kube • 390
Admiralty • 391
Liqo • 392
Introducing the Gardener project
Understanding the terminology of Gardener • 393
Understanding the conceptual model of Gardener • 394
Diving into the Gardener architecture • 395
Managing the cluster state • 395
Managing the control plane • 395
Preparing the infrastructure • 395
Using the Machine controller manager • 395
Networking across clusters • 396
Monitoring clusters • 396
The gardenctl CLI • 396
Extending Gardener • 397
Summary
Running
Running functions as a service on “serverless” infrastructure •
Azure AKS and Azure Container Instances • 406
AWS EKS and Fargate • 408
Google Cloud Run • 409
Knative
Knative serving • 410
Install a quickstart environment • 411
The Knative Service object • 413
Creating new revisions • 415
The Knative Route object • 415
The Knative Configuration object • 417
Knative eventing • 419
Getting familiar with Knative eventing terminology • 419
The architecture of Knative eventing • 421
Checking the scale to zero option of Knative • 422
Kubernetes Function-as-a-Service frameworks
OpenFaaS • 423
Delivery pipeline • 423
OpenFaaS features • 424
OpenFaaS architecture • 425
Taking OpenFaaS for a ride • 426
Fission • 436
Fission executor • 437
Fission workflows • 438
Experimenting with Fission • 441
Logging • 446
Log format • 446
Log storage • 446
Log aggregation • 446
Metrics • 447
Distributed tracing • 447
Application error reporting • 448
Dashboards and visualization • 448
Alerting • 448
Logging with Kubernetes
Container logs • 449
Kubernetes component logs • 450
Centralized logging • 450
Choosing a log collection strategy • 450
Cluster-level central logging • 452
Remote central logging • 452
Dealing with sensitive log information • 453
Using Fluentd for log collection • 453
Collecting metrics with Kubernetes
Monitoring with the Metrics Server • 455
The rise of Prometheus • 457
Installing Prometheus • 458
Interacting with Prometheus • 460
Incorporating kube-state-metrics • 461
Utilizing the node exporter • 462
Incorporating custom metrics • 463
Alerting with Alertmanager • 463
Visualizing your metrics with Grafana • 465
Considering Loki • 467
Distributed tracing with Kubernetes
What is OpenTelemetry? • 468
OpenTelemetry tracing concepts • 469
Introducing Jaeger • 469
Jaeger architecture • 470
Installing Jaeger • 471
Troubleshooting problems
Taking advantage of staging environments • 474
Detecting problems at the node level • 475
Problem daemons • 476
Dashboards vs. alerts • 476
Logs vs metrics vs. error reports • 477
Detecting performance and root cause with distributed tracing • 478 Summary
Chapter 14: Utilizing Service Meshes
Envoy • 484
Linkerd 2 • 484
Kuma • 484
AWS App Mesh • 484
Mæsh • 484
Istio • 485
OSM (Open Service Mesh) • 485
Cilium Service Mesh • 485
Envoy • 486
Pilot • 487
Citadel • 487
Galley • 488
Incorporating Istio into your Kubernetes cluster
Preparing a minikube cluster for Istio • 488
Installing Istio • 488
Installing BookInfo • 491 Working with
Traffic management • 494
Security • 497
Istio identity • 498
Istio certificate management • 499
Istio authentication • 499
Istio authorization • 500
Monitoring and observability • 503
Istio access logs • 503
Metrics • 506
Distributed tracing • 509
Visualizing your service mesh with Kiali • 512
Summary
Chapter 15: Extending Kubernetes 515
Working with the Kubernetes API
Understanding OpenAPI • 515
Setting up a proxy • 516
Exploring the Kubernetes API directly • 516
Using Postman to explore the Kubernetes API • 518
Filtering the output with HTTPie and jq • 519
Accessing the Kubernetes API via the Python client • 520
Dissecting the CoreV1Api group • 520
Listing objects • 523
Creating objects • 523
Watching objects • 525
Creating a pod via the Kubernetes API • 526
Controlling Kubernetes using Go and controller-runtime • 527
Using controller-runtime via go-k8s • 527
Invoking kubectl programmatically from Python and Go • 529
Using Python subprocess to run kubectl • 529
Extending the Kubernetes API
Understanding Kubernetes extension points and patterns • 533
Extending Kubernetes with plugins • 534
Extending Kubernetes with the cloud controller manager • 534
Extending Kubernetes with webhooks • 535
Extending Kubernetes with controllers and operators • 535
Extending Kubernetes scheduling • 536
Extending Kubernetes with custom container runtimes • 536
Introducing custom resources • 537
Developing custom resource definitions • 538
Integrating custom resources • 539
Dealing with unknown fields • 540
Finalizing custom resources • 542
Adding custom printer columns • 543
Understanding API server aggregation • 544
Building Kubernetes-like control planes • 545
Writing Kubernetes plugins
Writing a custom scheduler • 545
Understanding the design of the Kubernetes scheduler • 545
Scheduling pods manually • 548
Preparing our own scheduler • 549
Assigning pods to the custom scheduler • 550
Writing kubectl plugins • 551
Understanding kubectl plugins • 552
Managing kubectl plugins with Krew • 552
Creating your own kubectl plugin • 553
Employing access control webhooks
Using an authentication webhook • 555
Using an authorization webhook • 557
Using an admission control webhook • 559
Configuring a webhook admission controller on the fly • 560
Additional extension points ��������������������������������������������������������������������������������������������������
Providing custom metrics for horizontal pod autoscaling • 562
Extending Kubernetes with custom storage • 563
Summary
Chapter 16: Governing Kubernetes
Requirements of enterprise software • 566
Kubernetes and enterprise software • 566
Image management • 567
Pod security • 567
Network policy • 567
Configuration constraints • 568
RBAC and admission control • 568
Policy management • 568
Policy validation and enforcement • 568
Reporting • 569
Audit • 569
Admission control as the foundation of policy engines • 569
Responsibilities of a policy engine • 570
Quick review of open source policy engines • 570
OPA/Gatekeeper • 570
Kyverno • 571
jsPolicy • 572
Kubewarden • 572
Kyverno deep dive
Quick intro to Kyverno • 574
Installing and configuring Kyverno • 575
Installing pod security policies • 578
Configuring Kyverno • 579
Applying Kyverno policies • 580
Kyverno policies in depth • 583
Understanding policy settings • 583
Understanding Kyverno policy rules • 584
Validating requests • 587
Mutating resources • 588
Generating resources • 590
Advanced policy rules • 591
Writing and testing Kyverno policies • 592
Writing validating policies • 592
Writing mutating policies • 595
Writing generating policies • 597
Testing policies • 598
The Kyverno CLI • 598
Understanding Kyverno tests • 601
Writing Kyverno tests • 603
Running Kyverno tests • 605
Viewing Kyverno reports • 606
Deep integration • 614
Quotas and limits • 615
Real-world examples of quotas and limits • 615
Capacity planning • 616
When should you not use Managed Kubernetes? • 616
617
• 617
Hybrid • 617
Kubernetes on the edge • 618
Building effective processes for large-scale Kubernetes deployments ������������������������������������� 618
The development lifecycle • 618
Environments • 619
Separated environments • 619
Staging environment fidelity • 619
Resource quotas • 620
Promotion process • 620
Permissions and access control • 620
The principle of least privilege • 620
Assign permissions to groups • 620
Fine-tune your permission model • 620
Break glass • 621
Observability • 621
One-stop shop observability • 621
Troubleshooting your observability stack • 621
Handling infrastructure at scale ��������������������������������������������������������������������������������������������
Cloud-level considerations • 622
Compute • 622
Design your cluster breakdown • 623
Design your node pool breakdown • 623
Networking • 623
IP address space management • 623
Network topology • 624
Network segmentation • 624
Cross-cluster communication • 624
Cross-cloud communication • 625
Cross-cluster service meshes • 625
Managing egress at scale • 626
Managing the DNS at the cluster level • 627
Storage • 627
Choose the right storage solutions • 627
Data backup and recovery • 627
Storage monitoring • 628
Data security • 628
Optimize storage usage • 628
