Melanoma is one of the four major types of skin cancers caused by malignant growth in the melanocyte cells. It is the rarest one, accounting to only 1% of all skin cancer cases. However, it is the deadliest among all the skin cancer types. Owing to its rarity, efficient diagnosis of the disease becomes rather difficult. Here, a deep depthwise separable residual convolutional algorithm is introduced to perform binary melanoma classification on a dermoscopic skin lesion image dataset. Prior to training the model with the dataset noise removal from the images using non-local means filter is performed followed by enhancement using contrast-limited adaptive histogram equilisation over discrete wavelet transform algorithm. Images are fed to the model as multi-channel image matrices with channels chosen across multiple color spaces based on their ability to optimize the performance of the model. Proper lesion detection and classification ability of the model are tested by monitoring the gradient weighted class activation maps and saliency maps, respectively. Dynamic effectiveness of the model is shown through its performance in multiple skin lesion image datasets. The proposed model achieved an ACC of 99.50% on international skin imaging collaboration (ISIC), 96.77% on PH2, 94.44% on DermIS and 95.23% on MED-NODE datasets.