Skip Navigation
InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)SP
Posts
11
Comments
16
Joined
1 mo. ago
Kubernetes @programming.dev
SpiderUnderUrBed @lemmy.zip

How to see what is using flannel or circumvent flannel address usage in kubernetes?

I keep on getting issues with CNI and networking.. I just want my cluster to work.. anyways

 undefined
    
Apr 28 17:14:30 raspberrypi k3s[2373903]: time="2025-04-28T17:14:30+12:00" level=error msg="flannel exited: failed to register flannel network: failed to configure interface flannel.1: failed to set interface flannel.1 to UP state: address already in use"



  

How do i see what is using flannel Here is my server arguments:

 undefined
    
ExecStart=/usr/local/bin/k3s \
    server \
        --kubelet-arg=allowed-unsafe-sysctls=net.core.rmem_max,net.core.wmem_max,net.ipv4.ip_forward \
        --flannel-backend vxlan \
        --disable=traefik \
        --write-kubeconfig-mode 644


  

So I am using the default flannel backend, I tried repeatedly uninstalling then re-installing k3s, I deleted the current flannel interface with ip link, there is no other k3s instance is running, so why am I getting this issue?

  • Well I switched to cilium, same issue, and the reason I started using a CNI earlier than I intended was because flannel didn't work.

    This issue might seem complex but could you tell me some debugging stuff and logs to try to maybe get to the source of the issue or atleast provide a way to reproduce my issue (so I could maybe file a bug report)

  • Kubernetes @programming.dev
    SpiderUnderUrBed @lemmy.zip

    Memory issues with cilium despite plenty of memory being available

     spiderunderurbed@raspberrypi:~/k8s $ kubectl logs cilium-envoy-chzf8 -n kube-system
        
    external/com_github_google_tcmalloc/tcmalloc/system-alloc.cc:625] MmapAligned() failed - unable to allocate with tag (hint, size, alignment) - is something limiting address placement? 0x177840000000 1073741824 1073741824 @ 0x555b5fccc4 0x555b5f90e0 0x555b5f89a0 0x555b5d81d0 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614
    external/com_github_google_tcmalloc/tcmalloc/arena.cc:58] FATAL ERROR: Out of memory trying to allocate internal tcmalloc data (bytes, object-size); is something preventing mmap from succeeding (sandbox, VSS limitations)? 131072 632 @ 0x555b5fd034 0x555b5d8260 0x555b5f6694 0x555b5f6468 0x555b5cd988 0x555b4e3c84 0x555b4e09a0 0x7fb3918614
    spiderunderurbed@raspberrypi:~/k8s $ 
    
    
      

    Does anyone know how to fix the memory issue with cilium? or could link me to the docs or any issues about this. I just followed the instructions to install cilium, most stuff is

  •  undefined
        
    spiderunderurbed@raspberrypi:~/k8s $ kubectl get networkpolicy -A
    No resources found
    spiderunderurbed@raspberrypi:~/k8s $ 
    
    
      

    No networkpolicies.

     undefined
        
    spiderunderurbed@raspberrypi:~/k8s $ kubectl get pods -A | grep -i dns
    default                      pdns-admin-mysql-854c4f79d9-wsclq                         1/1     Running            1 (2d22h ago)    4d9h
    default                      pdns-mysql-master-6cddc8cd54-cgbs9                        1/1     Running            0                7h49m
    kube-system                  coredns-ff8999cc5-hchq6                                   1/1     Running            1 (2d22h ago)    4d11h
    kube-system                  svclb-pdns-mysql-master-1993c118-8xqzh                    3/3     Running            0                4d
    kube-system                  svclb-pdns-mysql-master-1993c118-whf5g                    3/3     Running            0                124m
    spiderunderurbed@raspberrypi:~/k8s $ 
    
    
      

    Ignore powerdns, its just extra stuff, but yeah coredns is running

     undefined
        
    spiderunderurbed@raspberrypi:~/k8s $  kubectl get endpoints  -n kube-system
    NAME             ENDPOINTS                                              AGE
    kube-dns         172.16.246.61:53,172.16.246.61:53,172.16.246.61:9153   4d11h
    metrics-server   172.16.246.45:10250                                    4d11h
    traefik          <none>                                                 130m
    spiderunderurbed@raspberrypi:~/k8s $ 
    
    
      

    ^ endpoints and services:

     undefined
        
    spiderunderurbed@raspberrypi:~/k8s $ kubectl get svc -n kube-system
    NAME             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
    kube-dns         ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       4d11h
    metrics-server   ClusterIP      10.43.67.112    <none>        443/TCP                      4d11h
    traefik          LoadBalancer   10.43.116.221   <pending>     80:31123/TCP,443:30651/TCP   131m
    spiderunderurbed@raspberrypi:~/k8s $ 
    
      
  • It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path.

  • https://pastebin.com/RhU5xtma I cant access any external address including dns servers, so, there is no firewall running on my pi (the master node), I can set the nameserver to be 1.1.1.1 in the pods config and iirc that works, but inside the pod, it doesn't work, so how do i fix this? You probably need more information so i can share. I am running calico as my CNI

  • Kubernetes @programming.dev
    SpiderUnderUrBed @lemmy.zip

    Traefik is not running properly, kube-apiserver pod might be down

    [EDIT] Soo.. kinda fixed? It was my backend, turns out, it forwards /nextcloud onto the nextcloud service, which does not know what to do with it unless I set something like site-url to include that path. So I made a middleware to strip the prefix, but now it cannot access any of its files because it will use the wrong path. I will look for siteurl settings but I dont think all of my services have one, so any advice would be appreciated for a general solution

    So currently my raspberrypi is connected to my internet under the ip, 192.168.68.77, (I configured traefik to work with that host and alternative hosts if need be). According to traefik logs I think that it does not work because it is missing access to the api server, although i could be wrong, i installed traefik via helm, and I have a config file for it, and disabled the default traefik given by k3s. here is the traefik config and logs: config: https://pastebin.com/XYH2LKF9 logs: https://pastebin.com/sbjPZCXv pods and svcs (al

  • Well, its kube-dns, and it simply, does not work, more specifically, it cannot resolve any external domains, I think it can resolve internal domains but I doubt thats working, but mainly it cant resolve external domains. I posted about it, here: https://lemmy.zip/post/36964791

    Recently, it was fixed because I found the correct endpoint, and uhh, now it stopped working, I updated the endpoint to the newer one, but it went back to the original issue detailed in that post.

  • No, i want to replace kube-dns and coredns, and some of my applications will resolve the ip at my dns server, then try those ips within the server, but mainly I want to replace the current dns stack due to several issues.

  • Kubernetes @programming.dev
    SpiderUnderUrBed @lemmy.zip

    How to get kubernetes to add all its internal dns entries to your own dns server

    By this I mean, I have a powerdns server running in my cluster, I would like Kubernetes to add/update dns entries in my dns server to reflect all services or any domains that would be used within the cluster, this is to fix a current issue I am having, and for general control and centralization purposes.

    Kubernetes @programming.dev
    SpiderUnderUrBed @lemmy.zip

    Kubernetes DNS broke

     spiderunderurbed@raspberrypi:~/k8s $ kubectl run -it --rm network-tools \
        
      --image=nicolaka/netshoot \
      --restart=Never \
      -- /bin/bash
    If you don't see a command prompt, try pressing enter.
    network-tools:~# cat /etc/resolv.conf
    search default.svc.cluster.local svc.cluster.local cluster.local
    nameserver 10.43.0.10
    options ndots:5
    network-tools:~# 
    
    
      

    DNS does not work in my k8s cluster. I dont know how to debug this, this is all my logs are in Coredns and kubedns:

     [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.override
        
    [WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
    
    
      

    This probably isnt enough, but what more can I do to debug this? I dont think its anything to do with my CNI, I am using calico, 1.1.1.1 as a nameserver or any nameserver works, but the issue is that internal to external dns mappings do not work, dns cannot resolve outside. Maybe not inside either according to this:

     undefined
        
    spiderunderurbed@raspberrypi:~/k8s $ 
      
    Linux Questions @lemmy.zip
    SpiderUnderUrBed @lemmy.zip

    Alternative applications for libvirt/kvm/qemu other than virt-manager (gui)?

    I find virt-manager hard to use, and not easily configurable, XML is the easiest but I don't always want to configure my vms in either a command-line or XML directly, is there any graphical alternative to virt-manager that uses the entire or part of the stack?

    Nix / NixOS @programming.dev
    SpiderUnderUrBed @lemmy.zip

    Dynamic VFIO does not work

    https://pastebin.com/30Bh23EV and this:

     undefined
        
      DVfio.configuration = {
            systemd.tmpfiles.rules = [
              "f /tmp/enable-vfio-switch 0644 spiderunderurbed users -"
            ];
            environment.variables = {
              KWIN_DRM_DEVICES = lib.mkForce "";
            };
            environment.extraInit = ''
                    export KWIN_DRM_DEVICES=$(${vfio}/bin/vfio)
            '';
      };
    
      

    So there is a issue with my configuration, so, you dont really need to understand nix, just like, look at my qemu hook script, its in plain sh, and the stuff above might be self explanitory, the issue is, my nvidia drivers are still being used, despite, setting KWIN_DRM_DEVICES to card0 so the logs of libvirtd looks something like this: https://pastebin.com/TaKrsY9S if setting kwin_drm_devices to my gpu card does not work, i dont know what does and can use help

    Linux Questions @lemmy.zip
    SpiderUnderUrBed @lemmy.zip

    selinux and tmpfs mount issue

     undefined
        
    [spiderunderurbed@daspidercave:~]$ distrobox enter debian
    Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "tmpfs" to rootfs at "/sys/fs/selinux": create mountpoint for /sys/fs/selinux mount: mkdirat /var/lib/docker/overlay2/21421daf7f99a368b01031a78a899d0a459f341e9e942698981d2499a9aa042c/merged/sys/fs/selinux: operation not permitted: unknown
    Error: failed to start containers: debian
    [ble: exit 1]
    
    
      

    How do i fix? This was created normally, the container, I dont know what more information to add

  • Thats weird, because I clearly have free space:

     undefined
        
    spiderunderurbed@raspberrypi:~ $ df -h /
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/mmcblk0p2  235G  184G   39G  83% /
    spiderunderurbed@raspberrypi:~ $ df -h .
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/mmcblk0p2  235G  184G   39G  83% /home
    spiderunderurbed@raspberrypi:~ $ 
    
    
      

    any idea of things I can try to fix/debug this?

  • Kubernetes @programming.dev
    SpiderUnderUrBed @lemmy.zip

    NodeNotReady despite no pressure and an available network kubernetes

    My cluster has been showing my raspberrypi node as "Ready" but according to the description of the node, the last log was "NodeNotReady" all debug guides say look for any pressure, like disk, pid, or so on, but there is no pressure, no absence of network. Here is the logs of my pi and pi status: https://pastebin.com/UULz6Hcy My pods are stuck in unknown (except jellyfin which is awaiting another node to come on): https://pastebin.com/vw2masAC A description of one of my pods if that helps: https://pastebin.com/s5W03s0E

    also i already tried re-installing k3s

    homelab @lemmy.ml
    SpiderUnderUrBed @lemmy.zip

    Cloudflare Tunnels is not working (DNS is broken)

    So I need help with a split dns approach, or a direct fix, normally when running my tunnel on the simplest configuration I get this error:

     undefined
        
    
    Couldn't resolve SRV record &{region1.v2.argotunnel.com. 7844 1 1}: lookup region1.v2.argotunnel.com. on 10.43.0.10:53: read udp 172.16.91.156:54443->10.43.0.10:53: i/o timeout
    
    
    
      

    When I tried to change the nameserver to cloudflare to make it accessible I get this error:

     undefined
        
    2025-04-07T10:06:38Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup traefik on 1.1.1.1:53: no such host" connIndex=3 event=1 ingressRule=3 originService=http://traefik/
    2025-04-07T10:06:38Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp: lookup traefik on 1.1.1.1:53: no such host" connIndex=3 dest=https://nextcloud.spidershomelab.xyz/index.php/204 event=0 ip=198.41.200.23
      
    homelab @lemmy.ml
    SpiderUnderUrBed @lemmy.zip

    Traefik: services can no longer access assets when stripping first prefix (how to set base url)

     undefined
        
    apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
      name: strip-first-prefix
      namespace: default
    spec:
    #  replacePathRegex:
    #    regex: "^/[^/]+(.*)"
    #    replacement: "$1"
      stripPrefix:
        prefixes:
          #- "/dashboard"
          #- "/api"
          - "/gitea"
          - "/wordpress"
          - "/vaultwarden"
          - "/pdns"
          - "/glance"
          - "/immich"
    
    
      

    So I have a issue. whenever I accessed all of my services via 192.168.1.22/wordpress for example. it forwarded that /wordpress to the actual wordpress domain, leading to page not found, however when i strip the initial proefix, i can access the base page, however, when lets say wordpress wants any css or assets, it will look at 192.168.1.22/assets which wont work, so basically, I need a way for sort of, emulate the url paths, so it wont take actual queries to places that dont exist and tries to access resources the incorrect way, i know siteURL exists for WP, but i want a catchall solution which helps my other services.