Kubernetes Plugin MariaDB related error
Summary
Reproducibility | Platform | OS | OS Version | Product Version |
---|---|---|---|---|
always | AMD64 | Debian | Bullseye | 13.0.1-22081215~bullseye |
Description
Backuping only the Kubernetes objects except Persistent Volume Claims (PVC) works like a charm. I've installed the Kubernetes plugin and the latest Bacula File Daemon on the master node (control plane) of our Kubernetes cluster. Bacula can access the Kubernetes cluster and backup every single object as YAML files.
The interesting part comes with trying to backup a PVC...
First of all I could build my own Bacula Backup Proxy Pod Image from the source and it's deployed into our local Docker image repository (repo). The Bacula File Daemon is configured properly I guess. Backup process started and the following things happened.
- Bacula File Daemon deployed Bacula Backup Proxy Pod Image into the Kubernetes cluster, so Bacula-backup container pod started.
- I got into the pod and I could see the Baculatar application started and running.
- The k8s_backend application started on the Bacula File Daemon host (kubernetes.server) in 2 instances.
- From the Bacula-backup pod I could check that Baculatar could connect to the k8s_backend at the default 9104 port (kubernetes.server:9104).
- I checked the console messages of the job with Bat that Bacula File Daemon started to process the configured PVC, started to write a pvc.tar but nothing happened.
- After default 600 sec, after timeout the job was cancelled.
Steps to Reproduce
- Setup a job with Kubernetes PVC backup.
- Start the backup job.
- The job will fail.
Additional Information
config files:
# cat /opt/bacula/etc/bacula-fd.conf
Director {
Name = bacula-dir
Password = "secret"
}
FileDaemon { # this is me
Name = bacula-fd
FDport = 9102 # where we listen for the director
WorkingDirectory = /opt/bacula/working
Pid Directory = /run/bacula
Maximum Concurrent Jobs = 20
Plugin Directory = /opt/bacula/plugins
FDAddress = node
PKI Signatures = Yes # Enable Data Signing
PKI Encryption = Yes # Enable Data Encryption
PKI Keypair = "/etc/bacula/certs/bacula-fd.pem" # Public and Private Keys
PKI Master Key = "/etc/bacula/certs/master.cert" # ONLY the Public Key
}
relevant part from Bacula Director config:
FileSet {
Name = "Kubernetes Set"
Include {
Options {
signature = SHA512
compression = GZIP
Verify = pins3
}
Plugin = "kubernetes: debug=1 baculaimage=repo/bacula-backup:04jan23 namespace=some pvcdata pluginhost=kubernetes.server timeout=120 verify_ssl=0 fdcertfile=/etc/bacula/certs/bacula-backup.cert fdkeyfile=/etc/bacula/certs/bacula-backup.key"
}
}
Error message from Bacula console:
bacula-fd kubernetes: Processing namespace: some kubernetes: Start backup volume claim: some-claim kubernetes: Prepare Bacula Pod on: node with: repo/bacula-backup:04jan23 kubernetes.server:9104 kubernetes: Connected to Kubernetes 1.25 - v1.25.4. bacula-sd Ready to append to end of Volume "Full-0513" size=1,680,733,693 node-fd Error: Read error on file /@kubernetes/namespaces/some/persistentvolumeclaims/some-claim.tar. ERR=Input/output error
Error: kubernetes: ConnectionServer: Timeout waiting...
Error: kubernetes: PTCOMM cannot get packet header from backend. bacula-sd Sending spooled attrs to the Director. Despooling 11,646 bytes ... node-fd Error: kubernetes: Unable to remove proxy Pod bacula-backup! Other operations with proxy Pod will fail! bacula-dir Fatal error: catreq.c:680 Restore object create error.
Error: Bacula Enterprise bacula-dir 13.0.1 (05Aug22): Build OS: x86_64-pc-linux-gnu-bacula-enterprise debian 11.2 JobId: 411957 Job: KubernetesBackup.2023-01-13_08.45.44_07 Backup Level: Full Client: "bacula-fd" 13.0.1 (05Aug22) x86_64-pc-linux-gnu-bacula-enterprise,debian,10.11 FileSet: "Kubernetes Set" 2023-01-13 08:39:12 Pool: "Full-Pool-Internal" (From Job FullPool override) Catalog: "MyCatalog" (From Client resource) Storage: "InternalStorageFull" (From Pool resource) Scheduled time: 13-Jan-2023 08:45:41 Start time: 13-Jan-2023 08:45:47 End time: 13-Jan-2023 08:51:51 Elapsed time: 6 mins 4 secs Priority: 10 FD Files Written: 34 SD Files Written: 0 FD Bytes Written: 76,496 (76.49 KB) SD Bytes Written: 0 (0 B) Rate: 0.2 KB/s Software Compression: 100.0% 1.0:1 Comm Line Compression: 2.6% 1.0:1 Snapshot/VSS: no Encryption: yes Accurate: yes Volume name(s): Full-0513 Volume Session Id: 559 Volume Session Time: 1672853724 Last Volume Bytes: 1,680,850,542 (1.680 GB) Non-fatal FD errors: 7 SD Errors: 0 FD termination status: OK SD termination status: SD despooling Attributes Termination: *** Backup Error *** Fatal error: sql_create.c:1273 Create db Object record INSERT INTO RestoreObject (ObjectName,PluginName,RestoreObject,ObjectLength,ObjectFullLength,ObjectIndex,ObjectType,ObjectCompression,FileIndex,JobId) VALUES ('RestoreOptions','kubernetes: debug=1 baculaimage=repo/bacula-backup:04jan23 namespace=some pvcdata pluginhost=kubernetes.server timeout=120 verify_ssl=0 fdcertfile=/etc/bacula/certs/bacula-backup.cert fdkeyfile=/etc/bacula/certs/bacula-backup.key','# Plugin configuration file\n# Version 1\nOptPrompt="K8S config file"\nOptDefault="None"\nconfig=@STR@\n\nOptPrompt="K8S API server URL/Host"\nOptDefault="None"\nhost=@STR@\n\nOptPrompt="K8S Bearertoken"\nOptDefault="None"\ntoken=@STR@\n\nOptPrompt="K8S API server cert verification"\nOptDefault="True"\nverify_ssl=@BOOL@\n\nOptPrompt="Custom CA Certs file to use"\nOptDefault="None"\nssl_ca_cert=@STR@\n\nOptPrompt="Output format when saving to file (JSON, YAML)"\nOptDefault="RAW"\noutputformat=@STR@\n\nOptPrompt="The address for listen to incoming backup pod data"\nOptDefault="FDAddress"\nfdaddress=@STR@\n\nOptPrompt="The port for opening socket for listen"\nOptDefault="9104"\nfdport=@INT32@\n\nOptPrompt="The endpoint address for backup pod to connect"\nOptDefault="FDAddress"\npluginhost=@STR@\n\nOptPrompt="The endpoint port to connect"\nOptDefault="9104"\npluginport=@INT32@\n\n',859,859,0,27,0,1,411957) failed. ERR=Data too long for column 'PluginName' at row 1 bacula-sd Elapsed time=00:06:04, Transfer rate=310 Bytes/second bacula-fd Error: kubernetes: Error closing backend. Err=Child exited with code 1
Error: kubernetes: PTCOMM cannot get packet header from backend.
After getting some help from Bacula-users mailinglist, it seems it's a MariaDB/MySQL related DB-issue.