We introduce an image denoising algorithm which utilizes a novel online dictionary learning procedure together with patch ordering. The developed algorithm employs both the non-local image processing power of patch ordering and the sequential patch-based update of online dictionary learning. The patch ordering process exploits the similarities between patches of a given image which are extracted from different locations. Joint processing of the ordered set of image patches facilitates the non-local image processing ability of the algorithm. The algorithm starts with the extraction of a maximally overlapped set of patches from the given noisy image. Then, the extracted patches are reordered by using a distance measure, and the 3D ordered patch cube is formed. The ordered patch cube is used sequentially to update an overcomplete dictionary. In each iteration, firstly the present patch is denoised using sparse coding over the current overcomplete dictionary. Secondly, the overcomplete dictionary is updated using the current image patch, and the dictionary is passed to the next iteration. We call this process as "on the fly denoising", because each patch is individually denoised using an instantaneously updated overcomplete dictionary. Patch ordering together with online dictionary learning ensures that the dictionary is adapted to different neighborhoods of patches in the patch cube. This adaptation of the dictionary to specialized local patch structures in the patch cube promises improved denoising performance when compared to dictionary learning algorithms devoid of such adaptation. Simulation results indicate that the introduced online method presents improved denoising performance in comparison to both online and batch dictionary learning algorithms from the literature while maintaining similar computational complexity.