This study investigates the determinants of financial stability in Iranian insurance companies, with particular emphasis on financial leverage, investment efficiency, risk management, and market structure characteristics. To this end, panel data for insurance firms listed on the Tehran Stock Exchange over the period 2017–2023 are employed, and a hybrid empirical framework combining fixed-effects panel regression and the random forest machine learning algorithm is implemented. Financial stability is proxied by the Z-score, while firm-specific variables are complemented by market concentration measures and macroeconomic conditions. The panel regression results indicate that financial leverage exerts a negative and statistically significant effect on financial stability, whereas investment efficiency, reinsurance utilization, firm size, and liquidity significantly enhance financial stability. In contrast, market concentration indicators and economic growth play a relatively limited role. The random forest findings corroborate these results and identify financial leverage, investment efficiency, and reinsurance as the most important predictors of financial stability. Moreover, partial dependence analysis reveals a nonlinear and threshold-type relationship between financial leverage and financial stability, such that beyond a critical level, further increases in debt lead to a sharp deterioration in financial resilience, while investment efficiency exhibits a monotonic and positive effect. The results suggest that the financial stability of insurance companies is driven primarily by the quality of managerial decisions regarding capital structure, asset allocation, and risk management, whereas market structure factors and macroeconomic variables play a secondary role.