Burn severity in forests is commonly assessed in the field with visual ordinal estimates such as the Composite Burn Index (CBI). However, how CBI (a composite of several individual field measures) relates to independent quantitative measures of burn severity (e.g. fire-caused tree mortality, surface charring) has not been widely tested. Here, we use field data from 315 plots in 14 fires in the north-western USA to ask: (1) how CBI relates to eight independent field measures of burn severity; and (2) how these relationships vary across gradients of pre-fire forest structure. Overall, CBI corresponded well with most independent field measures, but some measures of extreme burn severity (e.g. deep charring on trees and snags) were not captured by CBI. Additionally, some measures of canopy burn severity corresponded to lower CBI values in forests with larger average tree size (diameter and height) - potentially from decoupling of surface and canopy fire effects in stands with larger, fire-resistant trees. Our findings suggest continued broad utility of CBI, while highlighting how the correspondence of aggregate plot-level CBI to different measures of burn severity can vary with forest conditions. We also suggest considerations for broadening CBI to account for more extreme levels of burn severity.