Overview
-
term
-
name of k8s’ objects: can’t have underline
_ -
key name of helm’s value: can’t have dash
-
K8s
-
loop through all specific pod name
for pod in $(kubectl get po -o json | jq -r '.items[] | select(.metadata.name|contains("test")) | .metadata.name'); do echo $pod; kubectl logs $pod --all-containers | grep " 500 " ; done
# or replace jq, use :
| grep test |cut -d" " -f1
-
Scale replica
kubectl scale --replicas=3 deployment/replicaset/rs xyz
-
Check CPU/Memory usage
kubectl top pod podname --namespace=XXX
-
Debug node, check kubelet log of node
kubectl debug node/xxxx -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0
kubectl debug node/xxxx -it --image=ubuntu -n xxx
# recycle
kubectl delete pod xxx
-
Delete all Evicted pods
kubectl get pods -A | grep Evicted | awk '{print $1,$2,$4}' | xargs kubectl delete pod $2 -n $1
-
Bulk delete jobs
kubectl delete jobs --field-selector status.successful=0
-
Copy file in/out of pod to root
kubectl cp examplefile.zip xxpod-2133rfsdf:/examplefile.zip
-
Ext4 Folder is not empty
-
Err
[ERROR] --initialize specified but the data directory has files in it. Aborting.
-
Ans
args:
- "--ignore-db-dir=lost+found"
-
Get pod event
# work on kubectl v1.14 against a v1.11 API
kubectl get event --namespace abc-namespace --field-selector involvedObject.name=my-pod-zl6m6
kubectl describe event [POD_NAME] --namespace [POD's_NAMESPACE]
-
Create self-signed CA
kubectl create secret tls daas-tls --key daas.trendmicro.com.key --cert daas.trendmicro.com.crt
-
Install kubectl
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl # For user
sudo mv ./kubectl /usr/bin/kubectl # For root
kubectl version
-
Apply kubeconfig
-
Operate current context token
kubectl config use-context {contextName} # set current context
kubectl config current-context # get current context
-
Switch current namespace(context)
kubectl config set-context --current --namespace={my namespace}
-
Probe (liveness/readiness)
-
Keep running pod
command: ["ping", "-t", "google.com"]
command: ["/bin/sh", "-ec", "while true; do echo 'test'; sleep 5 ; done"]
-
Add command to deployment
apiVersion: v1
kind: Pod
metadata:
name: command-demo
labels:
purpose: demonstrate-command
spec:
containers:
- name: command-demo-container
image: debian
command: ["printenv"]
args: ["HOSTNAME", "KUBERNETES_PORT"]
restartPolicy: OnFailure
-
Https (k8s ingress TLS) “default backend 404” error
-
Ans: tls secret not correct, TLS not
-
Create TLS secret by file: .crt .key (Letsencrypt)
kubectl -n XX create secret tls tls-XXX \
--cert=signed.crt \
--key=domain.key
-
Create an base64 string for k8s Secret Opaque
echo -n 'password' | base64
# Or Notepad++ => MINE tool > Base64 Encode with Unix EOL
<--- sample file
apiVersion: v1
kind: Secret
metadata:
name: your-secrets
type: Opaque
data:
root-password: XXXXXXX
-
Clean key by patch
$ kubectl patch configmap myconfigmap --type=json -p='[{"op": "remove", "path": "/data/mykey"}]'
-
Rollback version
# List old
kubectl rollout history deployment/app
# Rollback to
kubectl rollout undo deployment/app --to-revision=2
-
Release pv (Persistent Volume) to be avaliable again. unbind, unbound PV
kubectl edit pv PV_NAME
# Remove spec.claimRef
# Or command:
kubectl patch pv {{PV_NAME}} --type=json -p='[{"op": "remove", "path": "/spec/claimRef"}]'
-
Search and get pod name
kubectl get pods -l app=my-app -o custom-columns=:metadata.name
-
kubectl Copy file into pod: error directory not exists or not found.
kubectl --kubeconfig=xxx cp {{filename}} {{namespace}}/{{pod}}:/{{filename}}
# {{filename}} is needed!!
-
Nginx sample
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: via-ingress.pentaidea.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
ports:
- port: 80
targetPort: 80
selector:
app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
-
CronJob sample – with script file
apiVersion: batch/v1
kind: CronJob
metadata:
name: jj-job
annotations:
version: <VERSION>
spec:
schedule: "*/30 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
volumes:
- name: scripts-volume
configMap:
name: scripts-configmap
defaultMode: 0777
containers:
- name: jj-job
image: curlimages/curl:latest
volumeMounts:
- mountPath: /tmp
name: scripts-volume
resources:
limits:
cpu: 400m
memory: 512Mi
command:
- /bin/sh
- -c
- /tmp/run-script.sh
restartPolicy: Never
nodeSelector:
beta.kubernetes.io/os: linux
-
CronJob sample
apiVersion: batch/v1
kind: CronJob
metadata:
name: jj-triggerjob
spec:
schedule: "* 6 * * *"
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
backoffLimit: 1
ttlSecondsAfterFinished: 3600
template:
spec:
containers:
- name: jj-triggerjob
image: curlimages/curl:latest
resources:
limits:
cpu: '200m'
memory: '256Mi'
requests:
cpu: '100m'
memory: '128Mi'
command:
- /bin/sh
- -c
- " echo \"Running trigger job\";
status_code=$(curl -ivs -o /dev/null -w \"%{http_code}\" -X GET https://www.google.com/ -H \"my-trace-id: $(cat /proc/sys/kernel/random/uuid)\");
echo $status_code;
if ! echo $status_code | grep -e \"200\" -e \"301\" ;
then
echo \"Failed with status code: $status_code\";
echo \"Done trigger job\";
exit 1;
fi;
echo \"Passed\";
echo \"Done trigger job\";
"
# imagepullsecrets:
# - name: XXAccount
restartPolicy: Never
nodeSelector:
kubernetes.io/os: linux
-
Warning:
-
always has ;
-
can’t use square brackets [ ] around with if echo $xxx | grep
-
Use “curl -o /dev/null -s -w \”%{http_code}\” ” to get status code
-
Other example
-
run bash
command: ["bash"]
args:
- -c
- echo "Hello world!" && sleep 5 && exit 42
-
run inline script
command:
- python3
- -c
- |
import os, sys
if os.environ.get("JOB_COMPLETION_INDEX") == "2":
sys.exit(0)
else:
sys.exit(1)
-
Unknown object type “nil”
-
error msg
error: error validating "/home/jj/deployment.yaml": error validating data: [ValidationError(Ingress.spec.tls[0].hosts): unknown object type "nil" in Ingress.spec.tls[0].hosts[0],
-
Solve: Fix wrong yaml format.
-
Apply private registry credential
kubectl create secret generic regcred \
--from-file=.dockerconfigjson={{path/to/.docker/config.json}} \
--type=kubernetes.io/dockerconfigjson
# Add to deployment (pod)
spec:
containers:
- name: xxx
image: xxx
imagePullSecrets:
- name: regcred
-
Deployment not updating after image updated, Force deployment rolling-update
spec:
template:
spec:
containers:
- image: xxx
imagePullPolicy: Always
-
Deployment not updating after configmap updated
-
Update label to trigger deployment rolling update
metadata:
labels:
configmap-version: 1
-
ConfigMap has a size limit of 1Mb, exceed error:
-
Msg: The ConfigMap “XXXXXXX” is invalid: metadata.annotations: Too long: must have at most 262144 characters
-
Use kubectl replace
-
Ref: https://github.com/coreos/prometheus-operator/issues/535#issuecomment-319936366
Helm
-
apply deployment error
-
error:
Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"partner"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable -
The spec.selector.matchLabels.app, can’t be changed/updated after first deploy.
K8s dashboard
-
Paste string to EXEC UI
-
ctrl-shift-v
-
Login issue: namespace change to default
-
Solution: type it back at uri
-
Login issue: exec into pod via Firefox will redirect back to k8s portal.
-
Sol: Use other browsers.
DockerHub
-
Always get handshake fail when `docker pull`
-
Solution:
Login DockerHub with the account which has no email address.
-
Download image fail: Authentication fail 401
-
Message:
(HTTP code 401) unexpected – {“message”:”Get https://registry-1.docker.io/v2/library/jenkins/manifests/2.46.3: unauthorized: incorrect username or password”}
Solution:
logging into Kitematic with your username instead of email.
-
Check the files permission in docker image
docker run –rm -ti –entrypoint sh jj/docker-stacks -c “ls -alF /usr/local/bin/”
Docker
-
Switch user
Dockerfile
---
FROM tw.registry.trendmicro.com/ik8s/win-dotnetcore-runtime:3.1-nanoserver-1809
USER "ContainerAdministrator"
-
Force delete pod
kubectl -n yyy delete pods xxxx --force --grace-period 0
-
Docker with GrayLog
-
Unable to start container by docker-compose
-
Msg: “UnixHTTPConnectionPool(host=’localhost’, port=None): Read timed out. (read timeout=60)”
-
Ans: ` sudo service docker restart`
-
[Character in Dockerfile]: ” will be split by space
in echo " xxx string " > file.txt
# result: file.txt
# xxx
# string
-
[Character in Dockerfile]: ” will be remove inside ‘ “xxx” ‘
echo ' "xxx string" ' > file.txt
# result: file.txt
# xxx string
-
[Cronjob] – Clean container&image daily at mid-night
# Clean container
0 0 * * * docker rm -f $(docker ps -aq)
# Clean image without baseImage
0 5 * * * docker image prune -f; docker rmi -f $(docker images | awk '/^[^m][^c][^r]*/{ print $3 }')
0 5 * * * docker rmi -f $(docker images | awk '$1 !~/ik8s/{ print $3 }')
0 5 * * * docker image prune -f --filter="dangling=true"; docker image prune -f --all --filter until=168h
# Clean all unused build cache
docker builder prune -a
# Clean all
docker system prune -a
# Clean image older than 48h
docker image prune -f --all --filter until=48h
# Clean dangling images
docker rmi $(sudo docker images -f "dangling=true" -q)
-
Not enough memory to start Docker on Windows
-
Modify `C:\Program Files\Docker\Docker\resources\MobyLinux.ps1` and change `$Memory = 512` MB as you want
-
Install with `sudo` but `docker run` without it, got error: “docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.35/containers/create: dial unix /var/run/docker.sock: connect: permission denied.See ‘docker run –help’.“.
sudo groupadd docker
sudo usermod -aG docker $USER # Add user into group
Ref: https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user
-
ERROR: Get https://registry-1.docker.io/v2/: remote error: tls: handshake failure
docker login # Yes, login first
-
Windows 10: Change docker images and any stuff to another folder, notice that the slash and case of character.
-- C:\ProgramData\docker\config\daemon.json --
{
"registry-mirrors": [],
"insecure-registries": [],
"debug": true,
"experimental": false,
"graph":"D:\\ProgramData\\docker"
}
-
docker: Error response from daemon: driver failed programming external connectivity on endpoint
-
Restart docker
-
The SPA default page middleware could not return the default page ‘/index.html’ because it was not found, and no other middleware handled the request.
-
mkdir /host_mnt/c: file exists.
-
Re-apply Shared Drive in docker-Desktop.
Docker compose
-
Setting up network mode (avoid IP not found)
version: '3.1'
services:
zookeeper-1:
image: zookeeper:3.4.13
container_name: zk
network_mode: bridge
-
Docker IP not match
# Check docker container IP
docker network inspect XXX
# Rebuild network (restart not working)
docker-compose down
docker-compose up
PS. docker-compose restart <- won't rebuild
-
ERROR: client version 1.22 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version
--- docker-compose.yml ---
version: '2.1'
...
GKE
-
Copy file in/out of pod to root
kubectl cp examplefile.zip xxpod-2133rfsdf:/examplefile.zip
Leave a Reply