I'll go there, i don't constantly post questions but I've just been recently having alot of issues with CNI's, I might just delete this post

How to see what is using flannel or circumvent flannel address usage in kubernetes?
I keep on getting issues with CNI and networking.. I just want my cluster to work.. anyways
undefined
Apr 28 17:14:30 raspberrypi k3s[2373903]: time="2025-04-28T17:14:30+12:00" level=error msg="flannel exited: failed to register flannel network: failed to configure interface flannel.1: failed to set interface flannel.1 to UP state: address already in use"
How do i see what is using flannel Here is my server arguments:
undefined
ExecStart=/usr/local/bin/k3s \ server \ --kubelet-arg=allowed-unsafe-sysctls=net.core.rmem_max,net.core.wmem_max,net.ipv4.ip_forward \ --flannel-backend vxlan \ --disable=traefik \ --write-kubeconfig-mode 644
So I am using the default flannel backend, I tried repeatedly uninstalling then re-installing k3s, I deleted the current flannel interface with ip link, there is no other k3s instance is running, so why am I getting this issue?
I increase envoys memory requirement to 1gb, did not fix the issue.
Well I switched to cilium, same issue, and the reason I started using a CNI earlier than I intended was because flannel didn't work.
This issue might seem complex but could you tell me some debugging stuff and logs to try to maybe get to the source of the issue or atleast provide a way to reproduce my issue (so I could maybe file a bug report)

Memory issues with cilium despite plenty of memory being available
spiderunderurbed@raspberrypi:~/k8s $ kubectl logs cilium-envoy-chzf8 -n kube-system
external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x177840000000 1073741824 1073741824 @ 0x555b5fccc4 0x555b5f90e0 0x555b5f89a0 0x555b5d81d0 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614 external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 632 @ 0x555b5fd034 0x555b5d8260 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614 spiderunderurbed@raspberrypi:~/k8s $
Does anyone know how to fix the memory issue with cilium? or could link me to the docs or any issues about this. I just followed the instructions to install cilium, most stuff is
Not k3d, just plain k3s
undefined
spiderunderurbed@raspberrypi:~/k8s $ kubectl get networkpolicy -A No resources found spiderunderurbed@raspberrypi:~/k8s $
No networkpolicies.
undefined
spiderunderurbed@raspberrypi:~/k8s $ kubectl get pods -A | grep -i dns default pdns-admin-mysql-854c4f79d9-wsclq 1/1 Running 1 (2d22h ago) 4d9h default pdns-mysql-master-6cddc8cd54-cgbs9 1/1 Running 0 7h49m kube-system coredns-ff8999cc5-hchq6 1/1 Running 1 (2d22h ago) 4d11h kube-system svclb-pdns-mysql-master-1993c118-8xqzh 3/3 Running 0 4d kube-system svclb-pdns-mysql-master-1993c118-whf5g 3/3 Running 0 124m spiderunderurbed@raspberrypi:~/k8s $
Ignore powerdns, its just extra stuff, but yeah coredns is running
undefined
spiderunderurbed@raspberrypi:~/k8s $ kubectl get endpoints -n kube-system NAME ENDPOINTS AGE kube-dns 172.16.246.61:53,172.16.246.61:53,172.16.246.61:9153 4d11h metrics-server 172.16.246.45:10250 4d11h traefik <none> 130m spiderunderurbed@raspberrypi:~/k8s $
^ endpoints and services:
undefined
spiderunderurbed@raspberrypi:~/k8s $ kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 4d11h metrics-server ClusterIP 10.43.67.112 <none> 443/TCP 4d11h traefik LoadBalancer 10.43.116.221 <pending> 80:31123/TCP,443:30651/TCP 131m spiderunderurbed@raspberrypi:~/k8s $
It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path.
https://pastebin.com/RhU5xtma I cant access any external address including dns servers, so, there is no firewall running on my pi (the master node), I can set the nameserver to be 1.1.1.1 in the pods config and iirc that works, but inside the pod, it doesn't work, so how do i fix this? You probably need more information so i can share. I am running calico as my CNI

Traefik is not running properly, kube-apiserver pod might be down
[EDIT] Soo.. kinda fixed? It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path. I will look for siteurl settings but I dont think all of my services have one, so any advice would be appreciated for a general solution
So currently my raspberrypi is connected to my internet under the ip, 192.168.68.77, (I configured traefik to work with that host and alternative hosts if need be). According to traefik logs I think that it does not work because it is missing access to the api server, although i could be wrong, i installed traefik via helm, and I have a config file for it, and disabled the default traefik given by k3s. here is the traefik config and logs: config: https://pastebin.com/XYH2LKF9 logs: https://pastebin.com/sbjPZCXv pods and svcs (al
Could you explain the steps to active your setup? I'm sure I can modify it for powerdns
Well, its kube-dns, and it simply, does not work, more specifically, it cannot resolve any external domains, I think it can resolve internal domains but I doubt thats working, but mainly it cant resolve external domains. I posted about it, here: https://lemmy.zip/post/36964791
Recently, it was fixed because I found the correct endpoint, and uhh, now it stopped working, I updated the endpoint to the newer one, but it went back to the original issue detailed in that post.
No, i want to replace kube-dns and coredns, and some of my applications will resolve the ip at my dns server, then try those ips within the server, but mainly I want to replace the current dns stack due to several issues.
Know anything that can do what I want?
I assume external dns means it will add all the domains like
<pod>
.<namespace>
.svc.cluster.local entries to that, or just<pod>
Any idea how I do that?

How to get kubernetes to add all its internal dns entries to your own dns server
By this I mean, I have a powerdns server running in my cluster, I would like Kubernetes to add/update dns entries in my dns server to reflect all services or any domains that would be used within the cluster, this is to fix a current issue I am having, and for general control and centralization purposes.
I solved the issue, the jellyfin pod for some reason was connecting to the wrong endpoint for the internal kube-dns service, I fixed that, and also made it use the internal pods FQDN and it works.

Kubernetes DNS broke
spiderunderurbed@raspberrypi:~/k8s $ kubectl run -it --rm network-tools \
--image=nicolaka/netshoot \ --restart=Never \ -- /bin/bash If you don't see a command prompt, try pressing enter. network-tools:~# cat /etc/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.43.0.10 options ndots:5 network-tools:~#
DNS does not work in my k8s cluster. I dont know how to debug this, this is all my logs are in Coredns and kubedns:
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
This probably isnt enough, but what more can I do to debug this? I dont think its anything to do with my CNI, I am using calico, 1.1.1.1 as a nameserver or any nameserver works, but the issue is that internal to external dns mappings do not work, dns cannot resolve outside. Maybe not inside either according to this:
undefined
spiderunderurbed@raspberrypi:~/k8s $
It does not work, as long as it goes to a cloudflare domain, the is a io timeout because of some DNS issue, any other suggestions?

Alternative applications for libvirt/kvm/qemu other than virt-manager (gui)?
I find virt-manager hard to use, and not easily configurable, XML is the easiest but I don't always want to configure my vms in either a command-line or XML directly, is there any graphical alternative to virt-manager that uses the entire or part of the stack?

Dynamic VFIO does not work
https://pastebin.com/30Bh23EV and this:
undefined
DVfio.configuration = { systemd.tmpfiles.rules = [ "f /tmp/enable-vfio-switch 0644 spiderunderurbed users -" ]; environment.variables = { KWIN_DRM_DEVICES = lib.mkForce ""; }; environment.extraInit = '' export KWIN_DRM_DEVICES=$(${vfio}/bin/vfio) ''; };
So there is a issue with my configuration, so, you dont really need to understand nix, just like, look at my qemu hook script, its in plain sh, and the stuff above might be self explanitory, the issue is, my nvidia drivers are still being used, despite, setting KWIN_DRM_DEVICES to card0 so the logs of libvirtd looks something like this: https://pastebin.com/TaKrsY9S if setting kwin_drm_devices to my gpu card does not work, i dont know what does and can use help

selinux and tmpfs mount issue
undefined
[spiderunderurbed@daspidercave:~]$ distrobox enter debian Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/sys/fs/selinux": create mountpoint for /sys/fs/selinux mount: mkdirat /var/lib/docker/overlay2/21421daf7f99a368b01031a78a899d0a459f341e9e942698981d2499a9aa042c/merged/sys/fs/selinux: operation not permitted: unknown Error: failed to start containers: debian [ble: exit 1]
How do i fix? This was created normally, the container, I dont know what more information to add
Thats weird, because I clearly have free space:
undefined
spiderunderurbed@raspberrypi:~ $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/mmcblk0p2 235G 184G 39G 83% / spiderunderurbed@raspberrypi:~ $ df -h . Filesystem Size Used Avail Use% Mounted on /dev/mmcblk0p2 235G 184G 39G 83% /home spiderunderurbed@raspberrypi:~ $
any idea of things I can try to fix/debug this?

NodeNotReady despite no pressure and an available network kubernetes
My cluster has been showing my raspberrypi node as "Ready" but according to the description of the node, the last log was "NodeNotReady" all debug guides say look for any pressure, like disk, pid, or so on, but there is no pressure, no absence of network. Here is the logs of my pi and pi status: https://pastebin.com/UULz6Hcy My pods are stuck in unknown (except jellyfin which is awaiting another node to come on): https://pastebin.com/vw2masAC A description of one of my pods if that helps: https://pastebin.com/s5W03s0E
also i already tried re-installing k3s
Cloudflare Tunnels is not working (DNS is broken)
So I need help with a split dns approach, or a direct fix, normally when running my tunnel on the simplest configuration I get this error:
undefined
Couldn't resolve SRV record &{region1.v2.argotunnel.com. 7844 1 1}: lookup region1.v2.argotunnel.com. on 10.43.0.10:53: read udp 172.16.91.156:54443->10.43.0.10:53: i/o timeout
When I tried to change the nameserver to cloudflare to make it accessible I get this error:
undefined
2025-04-07T10:06:38Z ERR error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup traefik on 1.1.1.1:53: no such host" connIndex=3 event=1 ingressRule=3 originService=http://traefik/ 2025-04-07T10:06:38Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup traefik on 1.1.1.1:53: no such host" connIndex=3 dest=https://nextcloud.spidershomelab.xyz/index.php/204 event=0 ip=198.41.200.23
Traefik: services can no longer access assets when stripping first prefix (how to set base url)
undefined
apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: strip-first-prefix namespace: default spec: # replacePathRegex: # regex: "^/[^/]+(.*)" # replacement: "$1" stripPrefix: prefixes: #- "/dashboard" #- "/api" - "/gitea" - "/wordpress" - "/vaultwarden" - "/pdns" - "/glance" - "/immich"
So I have a issue. whenever I accessed all of my services via 192.168.1.22/wordpress for example. it forwarded that /wordpress to the actual wordpress domain, leading to page not found, however when i strip the initial proefix, i can access the base page, however, when lets say wordpress wants any css or assets, it will look at 192.168.1.22/assets which wont work, so basically, I need a way for sort of, emulate the url paths, so it wont take actual queries to places that dont exist and tries to access resources the incorrect way, i know siteURL exists for WP, but i want a catchall solution which helps my other services.