Faster than real-time wildland fire simulators are being increasingly adopted by land managers to provide decision support for tactical wildfire management and assist with strategic risk planning. These simulators are typically based on simple forward rate-of-spread algorithms that were predominantly developed using observations of experimental fires. Given their operational use, it is important that fire simulators be assessed in terms of their performance for their intended use; predicting the spatial progression of wildfires. However, the conditions under which wildfires occur cannot be easily replicated experimentally. We describe and demonstrate a method for use in model development, whereby a dataset comprised of wildfire case-studies is used for evaluating the predictive performance of fire simulators. Two different versions of the model PHOENIX RapidFire were assessed, one incorporating a novel algorithm that accounts fine-scale spatial variation in landscape dryness. Evaluation was done by comparing simulator predictions against contemporaneous observations of 9 different wildfires that occurred in Australia. Performance was quantified using the sum of the Area Difference Indices—a measure of prediction overlap calculated for each prediction/observation pair. The two versions of the model performed similarly, with the newer version being marginally (but not statistically significantly) better when outcomes were summarised across all fires. Despite this, it did not perform better in all cases, with three of the 9 fires better predicted using the original model. Wildfire evaluation datasets were demonstrated to provide valuable feedback for model development, however the limited availability of data means power is lacking for detailed comparisons. With increasingly extreme weather conditions resulting from climate change, conditions under which wildfires occur are likely to continue to extend well beyond those under which fire models algorithms were developed. Consequently, the adoption of improved methods for collecting and utilising wildfire data is critical to ensure fire simulators retain relevance.