Skip to content

functional-loadbalancer master job permafailing #3460

Closed
Bug
#3624
Closed
functional-loadbalancer master job permafailing#3460
Bug
#3624

Description

@stephenfin
stephenfin
opened on Jul 15, 2025

The functional-loadbalancer (master) job has been permafailing since June 18th (though the last known success on the master branch was on June 4th). Looking at the logs, it seems Octavia is unable to spin up a VM. From the o-cw logs:

Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: WARNING octavia.controller.worker.v2.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait' (f7108ec7-6dc8-4bd0-97c9-4af3e778bbec) transitioned into state 'FAILURE' from state 'RUNNING'
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: 17 predecessors (most recent first):
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:   Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-create-retry-subflow_retry' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': [(1, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4764f80>}), (2, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff47472c0>}), (3, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4766b10>}), (4, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff47659d0>}), (5, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff47665d0>}), (6, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4764e30>}), (7, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4767b30>}), (8, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4766810>}), (9, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff47669c0>}), (10, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff47460f0>}), (11, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4766a50>}), (12, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4767350>}), (13, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff4767980>}), (14, {'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-wait': <taskflow.types.failure.Failure object at 0x7f2ff477c080>}), (15, {})]}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:   |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-compute-create-retry-subflow'
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:      |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-mark-amphora-booting-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': 'dbef970a-acfd-4495-9005-6c3628b4ba92', 'compute_id': 'dd0a22a4-355e-47c1-b29d-1a53ddc5ee18'}, 'provides': None}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:         |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-amphora-computeid' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': 'dbef970a-acfd-4495-9005-6c3628b4ba92', 'compute_id': 'dd0a22a4-355e-47c1-b29d-1a53ddc5ee18'}, 'provides': None}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:            |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-compute-create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': 'dbef970a-acfd-4495-9005-6c3628b4ba92', 'server_pem': 'gAAAAABocxaj1Tx0554tjwItwraoc0wFe2JX0I_Aelw93vyGTz8Oa6CqYlSb1kxREiYnT746Xy_wjaldYM75d3PdJAEmIDe3H5R_9CdD_T9myEpNSkk882IFuvEqP5RlKObNEGq0rzEo4HlyFj_izCUcg-ihJutCQgGp0EASPbuaDCYr7sGGwsVUnjpzdzdJ327hOuwCeUWP-xu9nXjscIYRaZWbc7pu__tDQchGd03gBVb2cRzcOnSiqb2YbVh6gfEjmBS20e_8vcG7AOOVuM0hzuAuOmGptcMPXlkRqBhml1m9ohR7CxbbvdnoIiMYLfmwbowUUEPYch483MOHre423WnXdwMLxnu-40IMXFUF_GI2hAs-r9LfO7SnXvjEwgYBQX7NkUepszAnnVA1IQsHaS7PN3GhyXUAKf6hDoNz0ZPMEDo6BeQPF2bLQx3zzSHXw8H03ME8z3efBmEASDVZwKaNtP_ugALEwz5o9uuqWKA5Q4OWnPI6OqcSQuhenZO1msx16iuo4DTfkRU5DHseQdF_nFN5bCq-lQvFkPMIGgoggkewZ5JrPY_9XASzEjObySYtmMI5a9stJPkadE-B2BafHxPtexCWTS1hZTgtK-jWJmMlYyq42vasGAg_S3Zot-6_ffjnU4FTYvv5Y9JtfAMYWa6vt20Z9aTqj5feOhDNvOH5OjEk4nq58UmOIhwuzXFiGdDKnOYB6NbTkGE15araO2g5yIQpQ8D6nkZLxCplgnQogTHaMNR-9xoNSMPg825byDjommpBhYB-RHgaL8okat6ytMJdeVrzGeEWvjVOzim0NHM1TGFrCPYpRNu8vPwn_F8d0nhBCQ6QvFeDZdKvnuZnKPxMVOh0tW8qOSmKNoms5RGisbKvw7Hl1cam7G3UwxK4vBBoGvZ_voTOhTvTqGLHb_fNpdJus8apXFCxLduxbD0OFhtmbE4JYjRZwNXbKcaSgh3FQ5xZZBzojlA74v0mtYvgXouKdgCGz5pb0pFR6E0Q0qGnxIgTgtoFA9V9TiQXGMnjLLGMZlfPL19XoYEnr2pRBWmh20_Q87KWF8F4HK_olWT_lRVKxdbdR4S8rmZXyrArs_MzKJrdtawGq9UL5wJ89QrvcB1Dahg5o8lfHjSTsTXPOIWuKl75lUX9Si1MRZNcNWHYB08AkRtCuZTmN0n05mK58-2U0pLXLLEJdQJmJ7XoV1kEmfiBjkOwbrO9N4r_ivju8rdjaN7hsTIZj4R0rnD4DFEjbPFxDbYmbdWCtDC2yN9fs--udCY7yDbiYB2vzEmhJUpVmQaVqXuQGBDILfrkGTGgyMkN8efAitD2QPII08dwqkNO0VIBXV12AmGUrHW2V_ozWftD2uk94ZdcWcdSRvmoOicaojUmsGGgIO-G7yv9eC-zml_rWubIZ09JOvDNevKl0jsIQifxRW0nfJ6GSBLL5yJOtOxoIXY1knZEehXXqxv0aS2TwSOP6eTaJoWSGGSJOQZobNdQYuq0khFNq8_tBIk-_M5ZoLfCVAoOjMP8Zf9eI9HbbmOzxQLExFA1XZDu9XcYbrfAfUNDBXjoblKfTtuPx0xvta__Biv66mf3q7HDwgBsvE45APorLOcQCPfQtJKOLT0HjjA3LQ-sU5VbexgqLRz8TkgqwaOVmHr42wjWXHwaaVc5xqXqAXBZIaGoh6J6zw62E7KIQfuiIY7AEddbEBCnKN-vzvXR7WtmB-ZW5PnAvTttqMQ2Ug5J6NcMeuI_je-iHsIXswko1LZJg_I9fHIZu1Ui-3lfnjA2dhUMkJpFS_whF_zrYLDAfgUd3lsYwGDP6_st02B_tYhCHWuD9X1HJOFJd-zh_ofJ-pNGdxilr5_rf_be1F4hB8zAegPjjCndhz9R8ElmgIa6nl_leGg60ERlbRB7b1iG0slJRWRRF4vqp6M7VrKQbkisF08Ufdh8rK0F_9XL8cEjGa4dGvVCxrFfZ9bTcDEB5vYVPNimsUFHnarZr7CipynkqewZbPObwG_7V5YVjgr8nQom4SRLywXdo5XZIO8UEfkoVeCPTvdaHptiNPS0VItE5tdgE4v0Ftqae7MfQzI-AAJ6ijBuevV7oV7NAQubSFLRD9EBJHbN7DrufAajwkshHJuShcg6mzLc_jq7h9CCADl00od5QBsTtpo4loR1gvH94fz29vBjsrIV-leelezQYCPCI-2B24sgjEQeXxsNCqsdG7e-mu1MrDvxNh8Wi5_Xy5UEYe1lMVcGzgEO8uHRhq2WQ_cuQHo2d_Rxseht7Ae3DjOLPVhWmKH2tUB1PSx05VgQMfo41QODm8CqYqAYiiyXxy8slB37wUdL3MgJt5L0Wdo_X2zx-24MegCdzW9DJWCBNBo9MJMTkItldK4bzC_bfPwS5sMj1rFr9Nn1-7SIVPAtbOtvnxgNl_M42V7V3GHQ0-sA0Z3uBL68ijk16-_dJt55iNBY00KWNNXbO8ehtb1-0zM-zBeFMZsLgI7BMxv94laykAXsgGjZEzvIcOl54OxEeKQVtvbHQNh7b16zYY_GnhB2nkesqOEZLe05QlVXbsxpsM207AmAif9l3FkkfOBtcN_93ylkQLlfEIpImt-ANY_DrXZqCz7bCJ6uY5EsH7Nmg6lfMQoWJss0q8CCzFdj91Pq-3ZPALLYWnNI9n5C1HMIo4fnwJqGF-LUPiC14_iNFeZsnjjb0dJJIPV-tubhvpDjsg8N9i3qxhOAdGzaM_G7IXEVbaJ8t-a1BSmXEtrMjwYavatet1qdAux1uKEX_L-mP2CSHW7FPxPk4J93R0B8zYUfvK_nLB8F3fs6boGfffRqjxnwli0Malelaih6Xfd4jSYDk1cffYxr7wlRXJV4X4AJ9yzqRuBMTm5RU2XqmTjJ6wYOxya2T7CYgt2E2YfdDRUz1Sv9MbMV7j93aByHOeDtbi06aXKIv5rlCd53J2FKUf18c35PwDqTxxo7c5kAV9k-uClVVQ4tqE1d4h8_FmafPq35Hja5FDjpvXvCbGBmaM8lcB5hMhsTXWayhBk14kUl5w3TIv6dBtvR06okq9T_FrzxYxqG31qpzO-5kv57av_-t-Z_DAr3v0SXKLm3_GGBlPLlHI76S9gTg6-IC51zeKxRKSsiYj_IXAE79mTRlpAtsbw6dSX0aAMaVlmoi0i5fWQIK2_2FcqoGTymzQpxrLHLl-Etqv4-dW5ky6-tpeQjjvidHGK2etdgnizKN5GpTEDnXElGWpwBHzuTvn-o6rGtFjDvj0cYGrYLNoCdaTsQ8BZEkukScqR7XZAzVLhpLq6UdLsKJvcArSjNPMlRAsW4kR8D3uUtaMrO_mTFNiupCq9V6nfWfUSI8LD1gZELWG191rSL_mhLrHqqglLFjY3P-HtjNoLpNCTxFFhHwAeKI81aGelWURXN6c622dqjP7B6chS6x9Wa-SHeo0iYhyxQwN_w8B0uhiAnC8wFlj-zgaE74fEBieYE1Qd7VYoVv64RINaw7VLXr7I7RWBz1JFKE4dzCG3xqEXFmM57fZsRUWjDRFSnDgWXnfSp4WTCVdJasb4xpGpJek-3TRUPkW1oCBVQCVLAMgYbjzRCnTOhWFOfrC9auDk_ryb5WHqvAwSMAl8af39kX81Nykiy5tmKd3aJDVSFM1W-Wf-3i59Q9yI1KurJWYxkJPuiTHl2vWBg9GeInGgSfCNrq27SsLPsUNcA0NQcz3xbDVhebyzVohIdF1qlPF7YtOhsOUDFt-TwAmXVO75qc-B5Ys4-DPntkc3TvGjxziT6qtvPzCy1wsVA339F5E8PdOvKlEs1ioWuqPuGromEtN-3zMstwZQKbdePsPM8e1jOHzzjaZZp1J1yV0SRmFY-HLPBT1yjO0s3r1NeskIvwWyVSetR4e8c-DX2r36Ltqw72nF9rzsBizAtTRuWwr9bpN7CMPEwynX3DrbKlLJvnFs0XaUvSrDJ_UaiN8C37L-Itu217C3CsibP-h47z0C6t9q3Qk4WUqJckDtOEyHm5BDuR1n3CsbRNRVWaMrYUE43', 'server_group_id': None, 'build_type_priority': 40, 'flavor': {}, 'availability_zone': {}}, 'provides': 'dd0a22a4-355e-47c1-b29d-1a53ddc5ee18'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:               |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': 'dbef970a-acfd-4495-9005-6c3628b4ba92', 'server_pem': 'gAAAAABocxaj1Tx0554tjwItwraoc0wFe2JX0I_Aelw93vyGTz8Oa6CqYlSb1kxREiYnT746Xy_wjaldYM75d3PdJAEmIDe3H5R_9CdD_T9myEpNSkk882IFuvEqP5RlKObNEGq0rzEo4HlyFj_izCUcg-ihJutCQgGp0EASPbuaDCYr7sGGwsVUnjpzdzdJ327hOuwCeUWP-xu9nXjscIYRaZWbc7pu__tDQchGd03gBVb2cRzcOnSiqb2YbVh6gfEjmBS20e_8vcG7AOOVuM0hzuAuOmGptcMPXlkRqBhml1m9ohR7CxbbvdnoIiMYLfmwbowUUEPYch483MOHre423WnXdwMLxnu-40IMXFUF_GI2hAs-r9LfO7SnXvjEwgYBQX7NkUepszAnnVA1IQsHaS7PN3GhyXUAKf6hDoNz0ZPMEDo6BeQPF2bLQx3zzSHXw8H03ME8z3efBmEASDVZwKaNtP_ugALEwz5o9uuqWKA5Q4OWnPI6OqcSQuhenZO1msx16iuo4DTfkRU5DHseQdF_nFN5bCq-lQvFkPMIGgoggkewZ5JrPY_9XASzEjObySYtmMI5a9stJPkadE-B2BafHxPtexCWTS1hZTgtK-jWJmMlYyq42vasGAg_S3Zot-6_ffjnU4FTYvv5Y9JtfAMYWa6vt20Z9aTqj5feOhDNvOH5OjEk4nq58UmOIhwuzXFiGdDKnOYB6NbTkGE15araO2g5yIQpQ8D6nkZLxCplgnQogTHaMNR-9xoNSMPg825byDjommpBhYB-RHgaL8okat6ytMJdeVrzGeEWvjVOzim0NHM1TGFrCPYpRNu8vPwn_F8d0nhBCQ6QvFeDZdKvnuZnKPxMVOh0tW8qOSmKNoms5RGisbKvw7Hl1cam7G3UwxK4vBBoGvZ_voTOhTvTqGLHb_fNpdJus8apXFCxLduxbD0OFhtmbE4JYjRZwNXbKcaSgh3FQ5xZZBzojlA74v0mtYvgXouKdgCGz5pb0pFR6E0Q0qGnxIgTgtoFA9V9TiQXGMnjLLGMZlfPL19XoYEnr2pRBWmh20_Q87KWF8F4HK_olWT_lRVKxdbdR4S8rmZXyrArs_MzKJrdtawGq9UL5wJ89QrvcB1Dahg5o8lfHjSTsTXPOIWuKl75lUX9Si1MRZNcNWHYB08AkRtCuZTmN0n05mK58-2U0pLXLLEJdQJmJ7XoV1kEmfiBjkOwbrO9N4r_ivju8rdjaN7hsTIZj4R0rnD4DFEjbPFxDbYmbdWCtDC2yN9fs--udCY7yDbiYB2vzEmhJUpVmQaVqXuQGBDILfrkGTGgyMkN8efAitD2QPII08dwqkNO0VIBXV12AmGUrHW2V_ozWftD2uk94ZdcWcdSRvmoOicaojUmsGGgIO-G7yv9eC-zml_rWubIZ09JOvDNevKl0jsIQifxRW0nfJ6GSBLL5yJOtOxoIXY1knZEehXXqxv0aS2TwSOP6eTaJoWSGGSJOQZobNdQYuq0khFNq8_tBIk-_M5ZoLfCVAoOjMP8Zf9eI9HbbmOzxQLExFA1XZDu9XcYbrfAfUNDBXjoblKfTtuPx0xvta__Biv66mf3q7HDwgBsvE45APorLOcQCPfQtJKOLT0HjjA3LQ-sU5VbexgqLRz8TkgqwaOVmHr42wjWXHwaaVc5xqXqAXBZIaGoh6J6zw62E7KIQfuiIY7AEddbEBCnKN-vzvXR7WtmB-ZW5PnAvTttqMQ2Ug5J6NcMeuI_je-iHsIXswko1LZJg_I9fHIZu1Ui-3lfnjA2dhUMkJpFS_whF_zrYLDAfgUd3lsYwGDP6_st02B_tYhCHWuD9X1HJOFJd-zh_ofJ-pNGdxilr5_rf_be1F4hB8zAegPjjCndhz9R8ElmgIa6nl_leGg60ERlbRB7b1iG0slJRWRRF4vqp6M7VrKQbkisF08Ufdh8rK0F_9XL8cEjGa4dGvVCxrFfZ9bTcDEB5vYVPNimsUFHnarZr7CipynkqewZbPObwG_7V5YVjgr8nQom4SRLywXdo5XZIO8UEfkoVeCPTvdaHptiNPS0VItE5tdgE4v0Ftqae7MfQzI-AAJ6ijBuevV7oV7NAQubSFLRD9EBJHbN7DrufAajwkshHJuShcg6mzLc_jq7h9CCADl00od5QBsTtpo4loR1gvH94fz29vBjsrIV-leelezQYCPCI-2B24sgjEQeXxsNCqsdG7e-mu1MrDvxNh8Wi5_Xy5UEYe1lMVcGzgEO8uHRhq2WQ_cuQHo2d_Rxseht7Ae3DjOLPVhWmKH2tUB1PSx05VgQMfo41QODm8CqYqAYiiyXxy8slB37wUdL3MgJt5L0Wdo_X2zx-24MegCdzW9DJWCBNBo9MJMTkItldK4bzC_bfPwS5sMj1rFr9Nn1-7SIVPAtbOtvnxgNl_M42V7V3GHQ0-sA0Z3uBL68ijk16-_dJt55iNBY00KWNNXbO8ehtb1-0zM-zBeFMZsLgI7BMxv94laykAXsgGjZEzvIcOl54OxEeKQVtvbHQNh7b16zYY_GnhB2nkesqOEZLe05QlVXbsxpsM207AmAif9l3FkkfOBtcN_93ylkQLlfEIpImt-ANY_DrXZqCz7bCJ6uY5EsH7Nmg6lfMQoWJss0q8CCzFdj91Pq-3ZPALLYWnNI9n5C1HMIo4fnwJqGF-LUPiC14_iNFeZsnjjb0dJJIPV-tubhvpDjsg8N9i3qxhOAdGzaM_G7IXEVbaJ8t-a1BSmXEtrMjwYavatet1qdAux1uKEX_L-mP2CSHW7FPxPk4J93R0B8zYUfvK_nLB8F3fs6boGfffRqjxnwli0Malelaih6Xfd4jSYDk1cffYxr7wlRXJV4X4AJ9yzqRuBMTm5RU2XqmTjJ6wYOxya2T7CYgt2E2YfdDRUz1Sv9MbMV7j93aByHOeDtbi06aXKIv5rlCd53J2FKUf18c35PwDqTxxo7c5kAV9k-uClVVQ4tqE1d4h8_FmafPq35Hja5FDjpvXvCbGBmaM8lcB5hMhsTXWayhBk14kUl5w3TIv6dBtvR06okq9T_FrzxYxqG31qpzO-5kv57av_-t-Z_DAr3v0SXKLm3_GGBlPLlHI76S9gTg6-IC51zeKxRKSsiYj_IXAE79mTRlpAtsbw6dSX0aAMaVlmoi0i5fWQIK2_2FcqoGTymzQpxrLHLl-Etqv4-dW5ky6-tpeQjjvidHGK2etdgnizKN5GpTEDnXElGWpwBHzuTvn-o6rGtFjDvj0cYGrYLNoCdaTsQ8BZEkukScqR7XZAzVLhpLq6UdLsKJvcArSjNPMlRAsW4kR8D3uUtaMrO_mTFNiupCq9V6nfWfUSI8LD1gZELWG191rSL_mhLrHqqglLFjY3P-HtjNoLpNCTxFFhHwAeKI81aGelWURXN6c622dqjP7B6chS6x9Wa-SHeo0iYhyxQwN_w8B0uhiAnC8wFlj-zgaE74fEBieYE1Qd7VYoVv64RINaw7VLXr7I7RWBz1JFKE4dzCG3xqEXFmM57fZsRUWjDRFSnDgWXnfSp4WTCVdJasb4xpGpJek-3TRUPkW1oCBVQCVLAMgYbjzRCnTOhWFOfrC9auDk_ryb5WHqvAwSMAl8af39kX81Nykiy5tmKd3aJDVSFM1W-Wf-3i59Q9yI1KurJWYxkJPuiTHl2vWBg9GeInGgSfCNrq27SsLPsUNcA0NQcz3xbDVhebyzVohIdF1qlPF7YtOhsOUDFt-TwAmXVO75qc-B5Ys4-DPntkc3TvGjxziT6qtvPzCy1wsVA339F5E8PdOvKlEs1ioWuqPuGromEtN-3zMstwZQKbdePsPM8e1jOHzzjaZZp1J1yV0SRmFY-HLPBT1yjO0s3r1NeskIvwWyVSetR4e8c-DX2r36Ltqw72nF9rzsBizAtTRuWwr9bpN7CMPEwynX3DrbKlLJvnFs0XaUvSrDJ_UaiN8C37L-Itu217C3CsibP-h47z0C6t9q3Qk4WUqJckDtOEyHm5BDuR1n3CsbRNRVWaMrYUE43'}, 'provides': None}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                  |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora_id': 'dbef970a-acfd-4495-9005-6c3628b4ba92'}, 'provides': 'gAAAAABocxaj1Tx0554tjwItwraoc0wFe2JX0I_Aelw93vyGTz8Oa6CqYlSb1kxREiYnT746Xy_wjaldYM75d3PdJAEmIDe3H5R_9CdD_T9myEpNSkk882IFuvEqP5RlKObNEGq0rzEo4HlyFj_izCUcg-ihJutCQgGp0EASPbuaDCYr7sGGwsVUnjpzdzdJ327hOuwCeUWP-xu9nXjscIYRaZWbc7pu__tDQchGd03gBVb2cRzcOnSiqb2YbVh6gfEjmBS20e_8vcG7AOOVuM0hzuAuOmGptcMPXlkRqBhml1m9ohR7CxbbvdnoIiMYLfmwbowUUEPYch483MOHre423WnXdwMLxnu-40IMXFUF_GI2hAs-r9LfO7SnXvjEwgYBQX7NkUepszAnnVA1IQsHaS7PN3GhyXUAKf6hDoNz0ZPMEDo6BeQPF2bLQx3zzSHXw8H03ME8z3efBmEASDVZwKaNtP_ugALEwz5o9uuqWKA5Q4OWnPI6OqcSQuhenZO1msx16iuo4DTfkRU5DHseQdF_nFN5bCq-lQvFkPMIGgoggkewZ5JrPY_9XASzEjObySYtmMI5a9stJPkadE-B2BafHxPtexCWTS1hZTgtK-jWJmMlYyq42vasGAg_S3Zot-6_ffjnU4FTYvv5Y9JtfAMYWa6vt20Z9aTqj5feOhDNvOH5OjEk4nq58UmOIhwuzXFiGdDKnOYB6NbTkGE15araO2g5yIQpQ8D6nkZLxCplgnQogTHaMNR-9xoNSMPg825byDjommpBhYB-RHgaL8okat6ytMJdeVrzGeEWvjVOzim0NHM1TGFrCPYpRNu8vPwn_F8d0nhBCQ6QvFeDZdKvnuZnKPxMVOh0tW8qOSmKNoms5RGisbKvw7Hl1cam7G3UwxK4vBBoGvZ_voTOhTvTqGLHb_fNpdJus8apXFCxLduxbD0OFhtmbE4JYjRZwNXbKcaSgh3FQ5xZZBzojlA74v0mtYvgXouKdgCGz5pb0pFR6E0Q0qGnxIgTgtoFA9V9TiQXGMnjLLGMZlfPL19XoYEnr2pRBWmh20_Q87KWF8F4HK_olWT_lRVKxdbdR4S8rmZXyrArs_MzKJrdtawGq9UL5wJ89QrvcB1Dahg5o8lfHjSTsTXPOIWuKl75lUX9Si1MRZNcNWHYB08AkRtCuZTmN0n05mK58-2U0pLXLLEJdQJmJ7XoV1kEmfiBjkOwbrO9N4r_ivju8rdjaN7hsTIZj4R0rnD4DFEjbPFxDbYmbdWCtDC2yN9fs--udCY7yDbiYB2vzEmhJUpVmQaVqXuQGBDILfrkGTGgyMkN8efAitD2QPII08dwqkNO0VIBXV12AmGUrHW2V_ozWftD2uk94ZdcWcdSRvmoOicaojUmsGGgIO-G7yv9eC-zml_rWubIZ09JOvDNevKl0jsIQifxRW0nfJ6GSBLL5yJOtOxoIXY1knZEehXXqxv0aS2TwSOP6eTaJoWSGGSJOQZobNdQYuq0khFNq8_tBIk-_M5ZoLfCVAoOjMP8Zf9eI9HbbmOzxQLExFA1XZDu9XcYbrfAfUNDBXjoblKfTtuPx0xvta__Biv66mf3q7HDwgBsvE45APorLOcQCPfQtJKOLT0HjjA3LQ-sU5VbexgqLRz8TkgqwaOVmHr42wjWXHwaaVc5xqXqAXBZIaGoh6J6zw62E7KIQfuiIY7AEddbEBCnKN-vzvXR7WtmB-ZW5PnAvTttqMQ2Ug5J6NcMeuI_je-iHsIXswko1LZJg_I9fHIZu1Ui-3lfnjA2dhUMkJpFS_whF_zrYLDAfgUd3lsYwGDP6_st02B_tYhCHWuD9X1HJOFJd-zh_ofJ-pNGdxilr5_rf_be1F4hB8zAegPjjCndhz9R8ElmgIa6nl_leGg60ERlbRB7b1iG0slJRWRRF4vqp6M7VrKQbkisF08Ufdh8rK0F_9XL8cEjGa4dGvVCxrFfZ9bTcDEB5vYVPNimsUFHnarZr7CipynkqewZbPObwG_7V5YVjgr8nQom4SRLywXdo5XZIO8UEfkoVeCPTvdaHptiNPS0VItE5tdgE4v0Ftqae7MfQzI-AAJ6ijBuevV7oV7NAQubSFLRD9EBJHbN7DrufAajwkshHJuShcg6mzLc_jq7h9CCADl00od5QBsTtpo4loR1gvH94fz29vBjsrIV-leelezQYCPCI-2B24sgjEQeXxsNCqsdG7e-mu1MrDvxNh8Wi5_Xy5UEYe1lMVcGzgEO8uHRhq2WQ_cuQHo2d_Rxseht7Ae3DjOLPVhWmKH2tUB1PSx05VgQMfo41QODm8CqYqAYiiyXxy8slB37wUdL3MgJt5L0Wdo_X2zx-24MegCdzW9DJWCBNBo9MJMTkItldK4bzC_bfPwS5sMj1rFr9Nn1-7SIVPAtbOtvnxgNl_M42V7V3GHQ0-sA0Z3uBL68ijk16-_dJt55iNBY00KWNNXbO8ehtb1-0zM-zBeFMZsLgI7BMxv94laykAXsgGjZEzvIcOl54OxEeKQVtvbHQNh7b16zYY_GnhB2nkesqOEZLe05QlVXbsxpsM207AmAif9l3FkkfOBtcN_93ylkQLlfEIpImt-ANY_DrXZqCz7bCJ6uY5EsH7Nmg6lfMQoWJss0q8CCzFdj91Pq-3ZPALLYWnNI9n5C1HMIo4fnwJqGF-LUPiC14_iNFeZsnjjb0dJJIPV-tubhvpDjsg8N9i3qxhOAdGzaM_G7IXEVbaJ8t-a1BSmXEtrMjwYavatet1qdAux1uKEX_L-mP2CSHW7FPxPk4J93R0B8zYUfvK_nLB8F3fs6boGfffRqjxnwli0Malelaih6Xfd4jSYDk1cffYxr7wlRXJV4X4AJ9yzqRuBMTm5RU2XqmTjJ6wYOxya2T7CYgt2E2YfdDRUz1Sv9MbMV7j93aByHOeDtbi06aXKIv5rlCd53J2FKUf18c35PwDqTxxo7c5kAV9k-uClVVQ4tqE1d4h8_FmafPq35Hja5FDjpvXvCbGBmaM8lcB5hMhsTXWayhBk14kUl5w3TIv6dBtvR06okq9T_FrzxYxqG31qpzO-5kv57av_-t-Z_DAr3v0SXKLm3_GGBlPLlHI76S9gTg6-IC51zeKxRKSsiYj_IXAE79mTRlpAtsbw6dSX0aAMaVlmoi0i5fWQIK2_2FcqoGTymzQpxrLHLl-Etqv4-dW5ky6-tpeQjjvidHGK2etdgnizKN5GpTEDnXElGWpwBHzuTvn-o6rGtFjDvj0cYGrYLNoCdaTsQ8BZEkukScqR7XZAzVLhpLq6UdLsKJvcArSjNPMlRAsW4kR8D3uUtaMrO_mTFNiupCq9V6nfWfUSI8LD1gZELWG191rSL_mhLrHqqglLFjY3P-HtjNoLpNCTxFFhHwAeKI81aGelWURXN6c622dqjP7B6chS6x9Wa-SHeo0iYhyxQwN_w8B0uhiAnC8wFlj-zgaE74fEBieYE1Qd7VYoVv64RINaw7VLXr7I7RWBz1JFKE4dzCG3xqEXFmM57fZsRUWjDRFSnDgWXnfSp4WTCVdJasb4xpGpJek-3TRUPkW1oCBVQCVLAMgYbjzRCnTOhWFOfrC9auDk_ryb5WHqvAwSMAl8af39kX81Nykiy5tmKd3aJDVSFM1W-Wf-3i59Q9yI1KurJWYxkJPuiTHl2vWBg9GeInGgSfCNrq27SsLPsUNcA0NQcz3xbDVhebyzVohIdF1qlPF7YtOhsOUDFt-TwAmXVO75qc-B5Ys4-DPntkc3TvGjxziT6qtvPzCy1wsVA339F5E8PdOvKlEs1ioWuqPuGromEtN-3zMstwZQKbdePsPM8e1jOHzzjaZZp1J1yV0SRmFY-HLPBT1yjO0s3r1NeskIvwWyVSetR4e8c-DX2r36Ltqw72nF9rzsBizAtTRuWwr9bpN7CMPEwynX3DrbKlLJvnFs0XaUvSrDJ_UaiN8C37L-Itu217C3CsibP-h47z0C6t9q3Qk4WUqJckDtOEyHm5BDuR1n3CsbRNRVWaMrYUE43'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                     |__Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39'}, 'provides': 'dbef970a-acfd-4495-9005-6c3628b4ba92'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                        |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                           |__Atom 'octavia.controller.worker.v2.tasks.network_tasks.GetSubnetFromVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': {'admin_state_up': True, 'description': 'TESTACCT-DESC-iPznMoRH', 'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'name': 'TESTACCT-ABiENYAj', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'vip_address': '192.168.1.48', 'vip_network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'vip_port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'vip_subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'vip_qos_policy_id': 'b3ab5134-6e48-4286-9d31-8a257a67d0ec', 'availability_zone': None}}, 'provides': {'id': '56bbe66c-68f1-402e-baff-c394addcf187', 'name': 'TESTACC-WomCi7JC', 'network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'gateway_ip': '192.168.1.1', 'cidr': '192.168.1.0/24', 'ip_version': 4, 'host_routes': []}}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                              |__Atom 'octavia.controller.worker.v2.tasks.network_tasks.UpdateVIPSecurityGroup' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39'}, 'provides': 'aacaa2d0-123f-46b4-bb87-5ec4879e8089'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                 |__Atom 'octavia.controller.worker.v2.tasks.database_tasks.UpdateAdditionalVIPsAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'additional_vips': []}, 'provides': {'admin_state_up': True, 'description': 'TESTACCT-DESC-iPznMoRH', 'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'name': 'TESTACCT-ABiENYAj', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'vip_address': '192.168.1.48', 'vip_network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'vip_port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'vip_subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'vip_qos_policy_id': 'b3ab5134-6e48-4286-9d31-8a257a67d0ec', 'availability_zone': None}}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                    |__Atom 'octavia.controller.worker.v2.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'vip': {'load_balancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'ip_address': '192.168.1.48', 'subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'load_balancer': None, 'qos_policy_id': None, 'octavia_owned': False, 'vnic_type': None, 'sg_ids': []}}, 'provides': {'admin_state_up': True, 'description': 'TESTACCT-DESC-iPznMoRH', 'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'name': 'TESTACCT-ABiENYAj', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'vip_address': '192.168.1.48', 'vip_network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'vip_port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'vip_subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'vip_qos_policy_id': 'b3ab5134-6e48-4286-9d31-8a257a67d0ec', 'availability_zone': None}}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                       |__Atom 'octavia.controller.worker.v2.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': {'admin_state_up': True, 'description': 'TESTACCT-DESC-iPznMoRH', 'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'name': 'TESTACCT-ABiENYAj', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'vip_address': '192.168.1.48', 'vip_network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'vip_port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'vip_subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'vip_qos_policy_id': 'b3ab5134-6e48-4286-9d31-8a257a67d0ec', 'availability_zone': None}}, 'provides': ({'load_balancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'ip_address': '192.168.1.48', 'subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'load_balancer': None, 'qos_policy_id': None, 'octavia_owned': False, 'vnic_type': None, 'sg_ids': []}, [])}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                          |__Atom 'reload-lb-before-allocate-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39'}, 'provides': {'admin_state_up': True, 'description': 'TESTACCT-DESC-iPznMoRH', 'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39', 'name': 'TESTACCT-ABiENYAj', 'project_id': 'a9a08b33abb14968a18bedc6629ebf72', 'vip_address': '192.168.1.48', 'vip_network_id': '3769cdf5-8dcc-4012-81c6-34f9d1881f73', 'vip_port_id': '31c54817-9ebb-4132-bd8e-e53b0bc13c16', 'vip_subnet_id': '56bbe66c-68f1-402e-baff-c394addcf187', 'vip_qos_policy_id': 'b3ab5134-6e48-4286-9d31-8a257a67d0ec', 'availability_zone': None}}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                             |__Atom 'octavia.controller.worker.v2.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': 'e6b959a8-bf1f-452b-8fe9-b4a2d9ed4a39'}, 'provides': None}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]:                                                |__Flow 'octavia-create-loadbalancer-flow': octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2025-07-13T02:15:28Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance dd0a22a4-355e-47c1-b29d-1a53ddc5ee18.', 'details': 'Traceback (most recent call last):\n  File "/opt/stack/nova/nova/conductor/manager.py", line 705, in build_instances\n    raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance dd0a22a4-355e-47c1-b29d-1a53ddc5ee18.\n'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker Traceback (most recent call last):
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker   File "/opt/stack/data/venv/lib/python3.12/site-packages/taskflow/engines/action_engine/executor.py", line 50, in _execute_task
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker     result = task.execute(**arguments)
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker              ^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker   File "/opt/stack/octavia/octavia/controller/worker/v2/tasks/compute_tasks.py", line 299, in execute
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker     raise exceptions.ComputeBuildException(fault=fault)
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2025-07-13T02:15:28Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance dd0a22a4-355e-47c1-b29d-1a53ddc5ee18.', 'details': 'Traceback (most recent call last):\n  File "/opt/stack/nova/nova/conductor/manager.py", line 705, in build_instances\n    raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance dd0a22a4-355e-47c1-b29d-1a53ddc5ee18.\n'}
Jul 13 02:15:29 fv-az1276-702 octavia-worker[115508]: ERROR octavia.controller.worker.v2.controller_worker

From the n-cpu log:

Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [None req-83cdbe52-45ee-4671-a07c-cabcbce6e833 admin admin] [instance: deb01d2a-6729-4a7b-822d-606e63f47563] Failed to build and run instance: libvirt.libvirtError: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: 2025-07-13T02:16:29.020977Z qemu-system-x86_64: -accel kvm: failed to initialize kvm: Permission denied
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563] Traceback (most recent call last):
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/compute/manager.py", line 2657, in _build_and_run_instance
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     self.driver.spawn(context, instance, image_meta,
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4864, in spawn
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     self._create_guest_with_network(
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 8273, in _create_guest_with_network
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     with excutils.save_and_reraise_exception():
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     self.force_reraise()
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     raise self.value
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 8251, in _create_guest_with_network
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     guest = self._create_guest(
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]             ^^^^^^^^^^^^^^^^^^^
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 8190, in _create_guest
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     guest.launch(pause=pause)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 167, in launch
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     with excutils.save_and_reraise_exception():
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     self.force_reraise()
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     raise self.value
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     return self._domain.createWithFlags(flags)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 186, in doit
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     rv = execute(f, *args, **kwargs)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 125, in execute
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     raise e.with_traceback(tb)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82, in tworker
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     rv = meth(*args, **kwargs)
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]          ^^^^^^^^^^^^^^^^^^^^^
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1415, in createWithFlags
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]     raise libvirtError('virDomainCreateWithFlags() failed')
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563] libvirt.libvirtError: internal error: process exited while connecting to monitor: Could not access KVM kernel module: Permission denied
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563] 2025-07-13T02:16:29.020977Z qemu-system-x86_64: -accel kvm: failed to initialize kvm: Permission denied
Jul 13 02:16:32 fv-az1276-702 nova-compute[80768]: ERROR nova.compute.manager [instance: deb01d2a-6729-4a7b-822d-606e63f47563]

Activity

  1. pierreprinetti
    added the
    Bug
    issue type on Jul 16, 2025

  2. kayrus commented on Jul 18, 2025

    @kayrus
    kayrus
    on Jul 18, 2025
    Contributor

    do we have an upstream octavia bug report?

  3. stephenfin commented on Jul 19, 2025

    @stephenfin
    stephenfin
    on Jul 19, 2025
    ContributorAuthor

    I don't think this is an Octavia bug: we're attempting to create a VM using KVM but the host doesn't support it. That points to an infra issue. I don't know why this isn't happening with e.g. the compute job though

  4. stephenfin
    changed the title [-]functional-loadbalancer master job perma-failing[/-] [+]functional-loadbalancer master job permafailing[/+] on Jul 21, 2025

  5. eshulman2 commented on Feb 2, 2026

    @eshulman2
    eshulman2
    on Feb 2, 2026
    Contributor

    Hi all, I was looking into this issue in ORC (as we are using a similar stage) the problem is a change in the amphora in 2025.2 which uses noble + amphora-apparmor causing it to fail when running nested. there are several options for solving this:

    1. Add a stage with a udev rule changing /dev/kvm permissions, something like:
        - name: Enable KVM access
      run: |
        echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
        sudo udevadm control --reload-rules
        sudo udevadm trigger --name-match=kvm
    1. Use an older amphora release OCTAVIA_AMP_DISTRIBUTION_RELEASE_ID=jammy
  6. mandre
    closed this as completedin #3624on Feb 2, 2026

  7. kayrus commented on Feb 9, 2026

    @kayrus
    kayrus
    on Feb 9, 2026
    Contributor

    @eshulman2 I wonder why didn't you pick the /dev/kvm permissions fix?

  8. eshulman2 commented on Feb 10, 2026

    @eshulman2
    eshulman2
    on Feb 10, 2026
    Contributor

    Hi @kayrus, the main reason I decided not go that route is to avoid masking potential issues in the future. By confining the fix to Octavia, we ensure that if a similar issue arises in another service, we can identify and investigate whether it’s actually related to gophercloud or something else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions