Skip to content

Problem: Tasks stuck in Waiting #3007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lug-gh opened this issue May 7, 2025 · 3 comments
Open

Problem: Tasks stuck in Waiting #3007

lug-gh opened this issue May 7, 2025 · 3 comments
Assignees
@mgukov
Labels
critical
Milestone
2.15

Comments

@lug-gh
Copy link
Contributor

lug-gh commented May 7, 2025

Issue

After upgrading from v2.13.12 to v2.14.8 I get the issue that after a short period of time my tasks don't start anymore. after restarting semaphore it works again for a few minutes, before it get's stuck again.
It's not specific to ansible etc., it affects every task type.

Impact

Ansible (task execution)

Installation method

Docker

Database

Postgres

Browser

No response

Semaphore Version

v2.14.8

Ansible Version



Logs & errors


Here's an example.

Task 5847 was started and executed, it failed (not because of semaphore, just a failed playbook because of the host).


After that task 5848 was started, but the task log kept being blank, nothing happened. After 4 minutes Task 5848 was stopped manually with the stop button.


Then the tasks 5849 and 5850 were triggered (in this case with the webhook integration, but it happens on any way, as you can see on task 5848). These tasks are still in "waiting", even if there's no other task running.


Image


Image


time="2025-05-07T10:44:08+02:00" level=info msg="Task 5847 added to queue"
time="2025-05-07T10:44:11+02:00" level=info msg="Set resource locker with TaskRunner 5847"
time="2025-05-07T10:44:11+02:00" level=info msg="Task 5847 removed from queue"
time="2025-05-07T10:45:02+02:00" level=info msg="Stopped running TaskRunner 5847"
time="2025-05-07T10:45:02+02:00" level=info msg="Release resource locker with TaskRunner 5847"
time="2025-05-07T10:46:04+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error
time="2025-05-07T10:46:04+02:00" level=error msg="write tcp 100.64.1.3:3000->100.64.1.4:49364: use of closed network connection" error="Cannot send close message"

time="2025-05-07T10:46:27+02:00" level=info msg="Task 5848 added to queue"
time="2025-05-07T10:46:31+02:00" level=info msg="Set resource locker with TaskRunner 5848"
time="2025-05-07T10:46:31+02:00" level=info msg="Task 5848 removed from queue"



time="2025-05-07T10:48:57+02:00" level=error msg="websocket: close sent" error="Cannot send close message"
time="2025-05-07T10:48:59+02:00" level=info msg="Receiving Integration from: 192.168.120.12"
time="2025-05-07T10:48:59+02:00" level=info msg="1 integrations found for alias *censored*"
time="2025-05-07T10:48:59+02:00" level=info msg="Running integration 2"
time="2025-05-07T10:49:02+02:00" level=info msg="Receiving Integration from: 192.168.120.12"
time="2025-05-07T10:49:02+02:00" level=info msg="1 integrations found for alias *censored*"
time="2025-05-07T10:49:02+02:00" level=info msg="Running integration 2"

time="2025-05-07T10:49:13+02:00" level=error msg="websocket: close sent" error="Cannot send close message"
time="2025-05-07T10:49:15+02:00" level=error msg="strconv.Atoi: parsing \"null\": invalid syntax" error="Bad request. Cannot get task_id from request"
2025/05/07 10:49:15 http: superfluous response.WriteHeader call from github.com/semaphoreui/semaphore/api/projects.GetTaskMiddleware.func1 (tasks.go:111)

time="2025-05-07T10:49:28+02:00" level=error msg="websocket: close sent" error="Cannot send close message"
time="2025-05-07T10:49:33+02:00" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"

time="2025-05-07T10:49:59+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error
time="2025-05-07T10:49:59+02:00" level=error msg="write tcp 100.64.1.3:3000->100.64.1.4:55434: use of closed network connection" error="Cannot send close message"

time="2025-05-07T10:50:14+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error
time="2025-05-07T10:50:14+02:00" level=error msg="write tcp 100.64.1.3:3000->100.64.1.4:45434: use of closed network connection" error="Cannot send close message"
time="2025-05-07T10:50:18+02:00" level=error msg="websocket: close sent" error="Cannot send close message"

time="2025-05-07T10:50:25+02:00" level=info msg="Stopped running TaskRunner 5848"
time="2025-05-07T10:50:25+02:00" level=info msg="Release resource locker with TaskRunner 5848"
time="2025-05-07T10:51:07+02:00" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"
time="2025-05-07T10:51:08+02:00" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"
time="2025-05-07T10:51:12+02:00" level=error msg="no rows in result set" error="Bad request. Cannot get task from database"

time="2025-05-07T10:51:25+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error

time="2025-05-07T10:51:25+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error
time="2025-05-07T10:51:25+02:00" level=error msg="write tcp 100.64.1.3:3000->100.64.1.4:55082: use of closed network connection" error="Cannot send close message"
time="2025-05-07T10:52:04+02:00" level=error msg="websocket: close sent" error="Cannot send close message"


time="2025-05-07T10:52:27+02:00" level=error msg="websocket: close 1006 (abnormal closure): unexpected EOF" fields.level=Error
time="2025-05-07T10:52:27+02:00" level=error msg="write tcp 100.64.1.3:3000->100.64.1.4:36484: use of closed network connection" error="Cannot send close message"



So I execute docker compose down && docker compose up -d , the tasks that were waiting are still waiting. If i stop them and then manually start it again, the task is executed immediately without any problem.


Manual installation - system information


No response


Configuration


No response


Additional information


No response

      		
    

      
  





        



            
            

              
            

        

      


      
    








      

    



      

  
      



  

  @lug-gh





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  Author


  


  

    

      

      
            lug-gh
  

      

      

      commented


        May 7, 2025


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
  
        
      Loading


  
    
  



    


  





      


        


  
    

      
          
I found out that it happens after a certain task. The special thing about the task is that it generates a lot of log lines in the semaphore (well over 100,000).

Maybe something is broken with the task log output? I have also noticed in other tasks that the log lines are thrown around, i.e. they are not in the correct order, e.g. this Terraform task:


Task 5873 added to queue
Started: 5873
Run TaskRunner with template: Proxmox automate Terraform

Preparing: 5873
Warning: Permanently added '[gitlab.example.org]' (ED25519) to the list of known hosts.
Updating Repository ssh://git@gitlab.example.org/example-devops/infrastructure/terraform/do-lab-cluster.git
- software-app-02 in module
- software-db in module
- software-gitlab in module
- software-web in module
- lab-jumphost in module
- example-lab-test-vm in module
Initializing provider plugins...
- Finding hashicorp/dns versions matching "3.4.2"...
- Finding tpretz/zabbix versions matching "0.17.0"...
- Finding hashicorp/http versions matching "3.4.5"...
- Finding telmate/proxmox versions matching "3.0.1-rc8"...
- Finding e-breuninger/netbox versions matching "3.10.0"...
- Using previously-installed hashicorp/http v3.4.5
- Using previously-installed telmate/proxmox v3.0.1-rc8
- Using previously-installed e-breuninger/netbox v3.10.0
- Using previously-installed hashicorp/dns v3.4.2
commands will detect it and remind you to do so if necessary.
module.example-lab-test-vm.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=example-lab-test-vm.lab.ha.example.org.]
module.software-web.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-web.lab.ha.example.org.]
module.software-gitlab.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-gitlab.lab.ha.example.org.]
module.lab-jumphost.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=lab-jumphost.lab.ha.example.org.]
module.software-db.data.netbox_tenant.netbox_tenant: Reading...
module.example-lab-test-vm.data.netbox_tenant.netbox_tenant: Reading...
module.example-lab-test-vm.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/107]
module.software-db.data.netbox_tenant.netbox_tenant: Read complete after 1s [id=34]
module.software-web.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/113]
module.software-app-01.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-app-02.data.netbox_tenant.netbox_tenant: Reading...
module.software-web.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.lab-jumphost.data.netbox_tenant.netbox_tenant: Reading...
module.example-lab-test-vm.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10862]
module.lab-jumphost.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10953]
module.lab-jumphost.netbox_virtual_machine.vm: Refreshing state... [id=211]
module.example-lab-test-vm.netbox_interface.vm_eth0: Refreshing state... [id=426]
module.software-db.netbox_interface.vm_eth0: Refreshing state... [id=495]
module.software-app-01.netbox_interface.vm_eth0: Refreshing state... [id=496]
module.software-web.netbox_interface.vm_eth0: Refreshing state... [id=494]
module.software-gitlab.netbox_interface.vm_eth0: Refreshing state... [id=484]
module.software-app-02.netbox_interface.vm_eth0: Refreshing state... [id=497]
module.lab-jumphost.netbox_interface.vm_eth0: Refreshing state... [id=489]
module.example-lab-test-vm.netbox_ip_address.vm_ip: Refreshing state... [id=1051]
module.software-db.netbox_ip_address.vm_ip: Refreshing state... [id=1120]
module.software-app-01.netbox_ip_address.vm_ip: Refreshing state... [id=1121]
module.software-web.netbox_ip_address.vm_ip: Refreshing state... [id=1119]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.example-lab-test-vm.proxmox_vm_qemu.vm will be updated in-place
  ~ resource "proxmox_vm_qemu" "vm" {
        id                     = "do-lab-pve1/qemu/107"
        name                   = "example-lab-test-vm"
        tags                   = "ha;terraform"
        # (55 unchanged attributes hidden)

      ~ disks {
          ~ scsi {
              ~ scsi0 {
                  ~ disk {
                        id                   = 0
                      ~ storage              = "local-lvm" -> "cephpool"
                      + iops_wr_concurrent   = 0
                      + mbps_r_burst         = 0
                      + mbps_r_concurrent    = 0
                      + mbps_wr_burst        = 0
                      + mbps_wr_concurrent   = 0
                      + replicate            = true
                      + size                 = "5G"
                      + storage              = "cephpool"
                    }
                }

                # (1 unchanged block hidden)
            }

            # (1 unchanged block hidden)
        }

        # (3 unchanged blocks hidden)
module.software-app-01.netbox_ip_address.vm_ip: Refreshing state... [id=1121]
module.software-gitlab.netbox_ip_address.vm_ip: Refreshing state... [id=1109]
module.software-web.netbox_ip_address.vm_ip: Refreshing state... [id=1119]
module.software-db.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=217]
module.software-app-02.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=216]
module.lab-jumphost.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=211]
module.example-lab-test-vm.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=147]
module.software-app-01.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=218]
module.software-gitlab.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=206]
module.software-web.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=219]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
                  ~ disk {
                        id                   = 0
                      ~ storage              = "local-lvm" -> "cephpool"
                        # (23 unchanged attributes hidden)
                    }
                }
              + scsi2 {
                  + disk {
                      + backup               = true
                      + discard              = true
                      + emulatessd           = true
                      + format               = "raw"
                      + iops_r_burst         = 0
                      + size                 = "5G"
                      + storage              = "cephpool"
                    }
                }

                # (1 unchanged block hidden)
            }

            # (1 unchanged block hidden)
        }

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.
module.example-lab-test-vm.proxmox_vm_qemu.vm: Modifying... [id=do-lab-pve1/qemu/107]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 40s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 50s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m0s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m10s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m20s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m30s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m10s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m20s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m30s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m40s elapsed]
Warning: Permanently added '[gitlab.example.org]' (ED25519) to the list of known hosts.
From ssh://gitlab.example.org/example-devops/infrastructure/terraform/do-lab-cluster
 * branch            semaphore  -> FETCH_HEAD
Already up to date.
Get current commit hash
Get current commit message
Initializing the backend...
Upgrading modules...
- software-app-01 in module
- Using previously-installed tpretz/zabbix v0.17.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
module.software-app-01.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-app1.lab.ha.example.org.]
module.software-app-02.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-app2.lab.ha.example.org.]
module.software-db.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-db.lab.ha.example.org.]
module.software-app-02.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve3/qemu/116]
module.software-gitlab.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/118]
module.software-app-01.data.netbox_tenant.netbox_tenant: Reading...
module.software-db.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/114]
module.lab-jumphost.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/120]
module.software-web.data.netbox_tenant.netbox_tenant: Reading...
module.software-app-01.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/115]
module.example-lab-test-vm.data.netbox_tenant.netbox_tenant: Read complete after 1s [id=33]
module.software-gitlab.data.netbox_tenant.netbox_tenant: Reading...
module.software-gitlab.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-db.netbox_virtual_machine.vm: Refreshing state... [id=217]
module.software-app-02.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-app-02.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10960]
module.software-gitlab.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10948]
module.software-db.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10961]
module.lab-jumphost.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=2]
module.software-web.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10958]
module.example-lab-test-vm.netbox_virtual_machine.vm: Refreshing state... [id=147]
module.software-app-01.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10959]
module.software-app-01.netbox_virtual_machine.vm: Refreshing state... [id=218]
module.software-web.netbox_virtual_machine.vm: Refreshing state... [id=219]
module.software-gitlab.netbox_virtual_machine.vm: Refreshing state... [id=206]
module.software-app-02.netbox_virtual_machine.vm: Refreshing state... [id=216]
module.software-gitlab.netbox_ip_address.vm_ip: Refreshing state... [id=1109]
module.software-app-02.netbox_ip_address.vm_ip: Refreshing state... [id=1122]
module.lab-jumphost.netbox_ip_address.vm_ip: Refreshing state... [id=1114]
module.example-lab-test-vm.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=147]
module.software-db.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=217]
module.software-app-01.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=218]
module.software-web.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=219]
module.software-gitlab.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=206]
module.software-app-02.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=216]
module.lab-jumphost.netbox_primary_ip.vm_primary_ip: Refreshing state... [id=211]

                        # (23 unchanged attributes hidden)
                    }
                }
              + scsi2 {
                  + disk {
                      + backup               = true
                      + discard              = true
                      + emulatessd           = true
                      + format               = "raw"
                      + iops_r_burst         = 0
                      + iops_r_burst_length  = 0
                      + iops_r_concurrent    = 0
                      + iops_wr_burst        = 0
                      + iops_wr_burst_length = 0
    }

Plan: 0 to add, 1 to change, 0 to destroy.

module.software-web.netbox_interface.vm_eth0: Refreshing state... [id=494]
module.software-db.netbox_ip_address.vm_ip: Refreshing state... [id=1120]
module.software-app-02.netbox_ip_address.vm_ip: Refreshing state... [id=1122]
module.lab-jumphost.netbox_ip_address.vm_ip: Refreshing state... [id=1114]
module.example-lab-test-vm.netbox_ip_address.vm_ip: Refreshing state... [id=1051]
  ~ update in-place

Terraform will perform the following actions:

  # module.example-lab-test-vm.proxmox_vm_qemu.vm will be updated in-place
  ~ resource "proxmox_vm_qemu" "vm" {
        id                     = "do-lab-pve1/qemu/107"
        name                   = "example-lab-test-vm"
        tags                   = "ha;terraform"
        # (55 unchanged attributes hidden)

      ~ disks {
          ~ scsi {
              ~ scsi0 {
                      + iops_r_burst_length  = 0
                      + iops_r_concurrent    = 0
                      + iops_wr_burst        = 0
                      + iops_wr_burst_length = 0
                      + iops_wr_concurrent   = 0
                      + mbps_r_burst         = 0
                      + mbps_r_concurrent    = 0
                      + mbps_wr_burst        = 0
                      + mbps_wr_concurrent   = 0
                      + replicate            = true
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 10s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 20s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 30s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m40s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 1m50s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m0s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 2m50s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Still modifying... [id=do-lab-pve1/qemu/107, 3m0s elapsed]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Modifications complete after 3m1s [id=do-lab-pve1/qemu/107]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
2025-05-07T10:04:45.941Z [ERROR] provider: error encountered while scanning stdout: error="read |0: file already closed"
module.software-app-02.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-app2.lab.ha.example.org.]
module.software-gitlab.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-gitlab.lab.ha.example.org.]
module.software-db.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-db.lab.ha.example.org.]
module.software-web.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-web.lab.ha.example.org.]
module.lab-jumphost.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=lab-jumphost.lab.ha.example.org.]
module.software-app-01.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=software-app1.lab.ha.example.org.]
module.example-lab-test-vm.dns_cname_record.vm_dns_cname_record: Refreshing state... [id=example-lab-test-vm.lab.ha.example.org.]
module.software-db.data.netbox_tenant.netbox_tenant: Reading...
module.example-lab-test-vm.data.netbox_tenant.netbox_tenant: Reading...
module.lab-jumphost.data.netbox_tenant.netbox_tenant: Reading...
module.software-app-02.data.netbox_tenant.netbox_tenant: Reading...
module.software-app-01.data.netbox_tenant.netbox_tenant: Reading...
module.software-gitlab.data.netbox_tenant.netbox_tenant: Reading...
module.software-web.data.netbox_tenant.netbox_tenant: Reading...
module.software-db.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.lab-jumphost.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/120]
module.software-db.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/114]
module.software-gitlab.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/118]
module.software-app-02.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-app-01.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve2/qemu/115]
module.example-lab-test-vm.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=33]
module.software-web.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/113]
module.software-app-02.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve3/qemu/116]
module.lab-jumphost.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=2]
module.software-gitlab.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-db.netbox_virtual_machine.vm: Refreshing state... [id=217]
module.example-lab-test-vm.proxmox_vm_qemu.vm: Refreshing state... [id=do-lab-pve1/qemu/107]
module.software-app-01.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.software-app-02.netbox_virtual_machine.vm: Refreshing state... [id=216]
module.software-db.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10961]
module.software-web.data.netbox_tenant.netbox_tenant: Read complete after 0s [id=34]
module.lab-jumphost.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10953]
module.software-gitlab.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10948]
module.software-app-01.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10959]
module.software-web.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10958]
module.software-app-02.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10960]
module.example-lab-test-vm.zabbix_host.vm_zabbix_agent: Refreshing state... [id=10862]
module.example-lab-test-vm.netbox_virtual_machine.vm: Refreshing state... [id=147]
module.lab-jumphost.netbox_virtual_machine.vm: Refreshing state... [id=211]
module.software-gitlab.netbox_virtual_machine.vm: Refreshing state... [id=206]
module.software-app-01.netbox_virtual_machine.vm: Refreshing state... [id=218]
module.software-web.netbox_virtual_machine.vm: Refreshing state... [id=219]
module.software-db.netbox_interface.vm_eth0: Refreshing state... [id=495]
module.software-app-02.netbox_interface.vm_eth0: Refreshing state... [id=497]
module.lab-jumphost.netbox_interface.vm_eth0: Refreshing state... [id=489]
module.example-lab-test-vm.netbox_interface.vm_eth0: Refreshing state... [id=426]
module.software-app-01.netbox_interface.vm_eth0: Refreshing state... [id=496]
module.software-gitlab.netbox_interface.vm_eth0: Refreshing state... [id=484]



Maybe both errors are somehow related, if not I can open another issue.

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
            

    

      
    

    


      

          @fiftin
fiftin




          added
  the 

  critical

 label


      May 7, 2025

    

  



    

    

      
    

    


      

          @fiftin
fiftin




          added this to the   2.14 milestone


      May 7, 2025

    

  








      

  
      



  

  @fiftin





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            fiftin
  

      

      

      commented


        May 7, 2025


      
    


  





      


        


  
    

      
          
Hi @lug-gh it is knows issue, fixing in progress.

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
            







      

  
      



  

  @fiftin





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  


  

    

      

      
            fiftin
  

      

      

      commented


        May 12, 2025


      
    


  





      


        


  
    

      
          
Hi @lug-gh Fix will be available in 2.15. Not in 2.14, too many changes.

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
            

    

      
    

    


      

          @fiftin
fiftin




          modified the milestones:
      2.14, 
      2.15


      May 12, 2025

    

  












  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

      

        

  
      @mgukov
  
    mgukov
    

      








      


  


  

    Labels
  



    

      

    critical









      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      
    
  
    2.15
  





    
      
          


  

    

      
        

          
    

    Development
  




            
    
No branches or pull requests







      
    

  




      

    

  
      

  

    

      3 participants
    

    

        
          @fiftin 
        
          @mgukov 
        
          @lug-gh