Cameras, especially those carried by drones, are the main tools used to detect wildfires in forests because cameras have much longer detection ranges than smoke sensors. Currently, deep learning is main method used for fire detection in images, and Transformer is the best algorithm. Swin Transformer restricts the computation to a fixed-size window, which reduces the amount of computation to a certain extent, but to allow pixel communication between windows, it adopts a shift window approach. Therefore, Swin Transformer requires multiple shifts to extend the receptive field to the entire image. This somewhat limits the network’s ability to capture global features at different scales. To solve this problem, instead of using the shift window method to allow pixel communication between windows, we downsample the feature map to the window size after capturing global features through a single Transformer, and we upsample the feature map to the original size and add it to the previous feature map. This way, there is no need for multiple layers of stacked window Transformers; global features are captured after each window Transformer operation. We conducted experiments on the Corsican fire dataset captured by ground cameras and on the Flame dataset captured by drone cameras. The results show that our algorithm performs the best. On the Corsican fire dataset, the mIoU, F1 score, and OA reached 79.4%, 76.6%, and 96.9%, respectively. On the Flame dataset, the mIoU, F1 score, and OA reached 84.4%, 81.6%, and 99.9%, respectively.