Red Flag Warnings (RFWs) issued by the National Weather Service in the United States (U.S.) are an important early warning system for fire potential based on forecasts of critical fire weather that promote increased fire activity, including the occurrence of large fires. However, verification of RFWs as they relate to fire activity is lacking, thereby limiting means to improve forecasts as well as increase value for end users. We evaluated the efficacy of RFWs as forecasts of large fire occurrence for the Northwestern U.S.—RFWs were shown to have widespread significant skill and yielded an overall 124% relative improvement in forecasting large fire occurrences than a reference forecast. We further demonstrate that the skill of RFWs is significantly higher for lightning-ignited large fires than for human-ignited fires and for forecasts issued during periods of high fuel dryness than those issued in the absence of high fuel dryness. The results of this first verification study of RFWs related to actualized fire activity lay the groundwork for future efforts towards improving the relevance and usefulness of RFWs and other fire early warning systems to better serve the fire community and public.