Abstract
As privacy concerns grow, computer vision must move beyond RGB, but can depth-only data compete? While most research focuses on encrypted color images or custom architectures, we ask: can off-the-shelf models classify objects using just depth?
We evaluate standard CNNs on depth-only inputs from two datasets: depth estimations from ImageNet using the Marigold model, and real-world depth scans from the Washington RGB-D dataset. Fine-tuning these models yielded promising results, with accuracies exceeding 70% (Top-5) on ImageNet subsets and 90% (Top-1) on Washington RGB-D.
These findings challenge the reliance on RGB data and specialized architectures for privacy-preserving vision. Depth data, whether approximated or raw, can enable strong classification performance using widely available models, opening doors to more complex applications like object detection and anomaly detection.
Introduction
Computer vision is highly advanced and can match or surpass human-level performance on tasks from image classification to object detection and segmentation. Most of these models rely on RGB images, which puts privacy at risk in settings such as hospitals, retirement homes, and child or elderly care centers, where protecting vulnerable people is critical.
We explore a promising alternative: depth images. They are captured by 3D cameras and keep the spatial structure needed for visual recognition without capturing identifiable appearance.
Our experiments focus on image classification, which we regard as the building block for the broader range of computer vision tasks, and so lay a foundation for future work.
Methodology
We investigate whether standard convolutional neural networks (CNNs) can classify objects from depth-only inputs.
Architectures. We evaluate four well-known CNNs, originally developed for RGB classification, to cover a range of depth and design:
- AlexNet: a relatively shallow model with fewer parameters, representing the early generation of deep learning classifiers.
- VGG19: a deeper model with a simple design based on stacked 3×3 convolutions.
- ResNet50: a 50-layer residual network with identity shortcut connections.
- Inception-v3: parallel convolutional paths and factorized filters to balance efficiency and performance.
Training regimes. Each model is evaluated in three configurations:
- Baseline: the original pre-trained weights, with no adaptation to depth.
- Partial fine-tuning: only the final layers are trainable.
- Complete fine-tuning: all parameters are trainable.
Evaluation. We report Top-1 accuracy (the correct label is the highest-ranked prediction) and Top-5 accuracy (it is among the five highest). Baselines are run only on ImageNet 1k-class depth. Both fine-tuning regimes are applied where there is enough data (ImageNet 1k-class and 200-class depth), while the smaller Washington RGB-D dataset uses partial fine-tuning only, to avoid overfitting. Fine-tuning uses an 80/20 train/validation split.
Datasets
Because of computational limits, we use only the validation portion of each dataset.
- ImageNet 1k-class depth. The 50,000-image ImageNet validation set. ImageNet has only RGB images, so we generated depth maps with Marigold, a diffusion-based depth estimator that produces single-channel 16-bit PNGs.
- ImageNet 200-class depth. A balanced subset of 200 randomly sampled classes, 10,000 images in total (50 per class), for lighter and more detailed experiments.
- Washington RGB-D. A real-world dataset captured with a Microsoft Kinect: 51 categories of everyday indoor objects across 300 instances. We use its 16-bit depth crops, 41,877 validation images, as they are.
Preprocessing. CNNs pre-trained on RGB expect 3-channel, 8-bit inputs, while depth is a single 16-bit channel. We min-max normalise the depth values to 8 bits, then expand them to three channels either by simply repeating the channel (“stacked”) or by applying a perceptually uniform colormap (Grayscale, Viridis, Plasma or Magma). We avoid the common jet (rainbow) colormap, which can distort the ordering of depth values.
Experiments and results
Baseline on ImageNet 1k-class depth
The models were pre-trained on the same 1,000 classes, so they can be evaluated as they are, with no fine-tuning. We compared the colormaps (Top-1 / Top-5 accuracy, %):
| Model | Stacked | Grayscale | Viridis | Plasma | Magma |
|---|---|---|---|---|---|
| AlexNet | 4.53 / 11.63 | 4.50 / 11.62 | 1.04 / 3.83 | 1.24 / 3.98 | 2.54 / 7.59 |
| VGG19 | 10.04 / 22.00 | 9.97 / 21.79 | 4.64 / 11.87 | 4.29 / 10.95 | 6.45 / 15.31 |
| ResNet50 | 13.18 / 28.16 | 13.27 / 28.16 | 6.56 / 16.23 | 5.83 / 14.00 | 9.71 / 21.44 |
| InceptionV3 | 16.48 / 33.05 | 16.40 / 32.94 | 13.16 / 26.66 | 12.36 / 25.34 | 15.01 / 30.07 |
Grayscale gave the results most familiar to the models, and simply repeating the depth channel three times (“stacked”) was as robust as applying a colormap. We therefore used the stacked encoding for all fine-tuning.
Fine-tuning on ImageNet 1k-class depth
In partial fine-tuning we froze everything except a few layers: the classifier for AlexNet and VGG19, the last residual block and the final layer for ResNet50, and the last inception block, the auxiliary classifier and the final layer for InceptionV3. In complete fine-tuning, all parameters were trainable.
| Model | Baseline | Partial fine-tuning | Complete fine-tuning |
|---|---|---|---|
| AlexNet | 4.53 / 11.63 | 18.05 / 36.41 | 23.62 / 45.76 |
| VGG19 | 10.04 / 22.00 | 25.40 / 48.18 | 34.20 / 60.96 |
| ResNet50 | 13.18 / 28.16 | 32.36 / 57.87 | 44.95 / 71.90 |
| InceptionV3 | 16.48 / 33.05 | 22.31 / 44.57 | 48.53 / 74.53 |
Both strategies improved accuracy a lot. Complete fine-tuning raised accuracy by almost 3× on average over the baseline, and InceptionV3 reached the best result (48.53% Top-1, 74.53% Top-5). The relative ranking of the models matches their ranking on RGB tasks, which suggests the gains come from the architecture rather than from anything specific to depth. The small number of examples per class limits overall performance.
ImageNet 200-class subset
Complete fine-tuning beats partial fine-tuning for every model, and the ranking is consistent with the 1,000-class results. Deeper networks gain the most from end-to-end adaptation when data is limited.
| Model | Partial fine-tuning | Complete fine-tuning |
|---|---|---|
| AlexNet | 26.25 / 52.90 | 26.40 / 51.25 |
| VGG19 | 32.50 / 59.95 | 36.70 / 65.25 |
| ResNet50 | 44.30 / 71.15 | 47.75 / 73.20 |
| InceptionV3 | 38.70 / 64.15 | 47.95 / 72.80 |
Washington RGB-D (real depth scans)
This dataset has only 51 classes but about 821 images per class, so the task is easier and there is far more training data. With partial fine-tuning alone, every model reached close to 90% Top-1, and VGG19 reached 92.23%. We skipped complete fine-tuning because the gain would not justify the cost: with enough data, transfer learning is already very effective.
| Model | Top-1 | Top-5 |
|---|---|---|
| AlexNet | 87.51 | 98.79 |
| VGG19 | 92.23 | 99.61 |
| ResNet50 | 88.48 | 99.31 |
| InceptionV3 | 89.45 | 99.08 |
Conclusion and future work
Depth-only images can give competitive classification performance with standard CNNs pre-trained on RGB, on both synthetic and real-world data, even without colour or fine texture.
- High accuracy on real data: VGG19 passed 92% Top-1 on Washington RGB-D, which supports depth-only inputs in practical, privacy-sensitive settings.
- Existing architectures work: with minimal preprocessing (normalisation, channel stacking, colormap mapping), standard models run on depth without changing their structure.
- Fine-tuning helps: even when limited to later layers, it consistently improved accuracy, most clearly on the 1,000-class task, where baseline accuracy more than doubled.
- Privacy-aware by nature: depth images lack identifying details such as faces or clothing patterns, which suits healthcare, education and public spaces.
Future work. The main limitation is the small number of depth samples per class, because large depth-only classification datasets are scarce. We plan to scale to the full ImageNet with estimated depth maps. We also want to try learned depth encodings with lightweight autoencoders instead of fixed colormaps, and to extend the work from classification to semantic segmentation, where many datasets already include depth.
Read the paper
The full paper, with references, is belowopens as a PDF with the button below. The abstract, figures and tables above are taken from it.