Since different regions of a color image generally exhibit different spectral characteristics, the energy compaction of applying a single spectral transform to all regions is largely inefficient from a compression perspective. Thus, it is proposed that different subsets of wavelet coefficients of a color image be subjected to different spectral transforms before the resultant coefficients are coded by an efficient wavelet coefficient coding scheme such as that used in JPEG2000 or color set partitioning in hierarchical trees (CSPIHT). A quadtree represents the spatial partitioning of the set of high frequency coefficients of the color planes into spatially oriented subsets which may be further partitioned into smaller directionally oriented subsets. The partitioning decisions and decisions to employ fixed or signal-dependent bases for each subset are rate-distortion (R-D) optimized by employing a known analytical R-D model for these coefficient coding schemes. A compression system of asymmetric complexity, that integrates the proposed adaptive spectral transform with the CSPIHT coefficient coding scheme yields average coding gains of 0.3 dB and 0.9 dB in the Y component at 1.0 b/p and 2.5 b/p, respectively, and 0.9 dB and 1.35 dB in the U and V components at 1.0 b/p and 2.5 b/p, respectively, over a reference compression system that integrates the single spectral transform derived from the entire image with the CSPIHT coefficient coding scheme.