We present a new image classification system by integrating color information and the extreme learning machine algorithm based on the localized soft-assignment coding classification framework. Our system is original with its unique structure. It improves the image classification accuracy significantly compared to the existing image classification systems. Specifically, on classification accuracy, our system can improve more than 4% on the Caltech-101 dataset and achieve up to 3% improvement on the Caltech-256 dataset compared to existing systems.