Previous studies observed that neural network models develop numerosity-selective units when trained to perform object classification, without explicit training on numerosity. However, the emergentist view was challenged by the finding that selectivity disappears with larger sample sizes for model evaluation. Here, we investigate whether this finding was due to the qualitative visual mismatch between training and evaluation data. We present experiments with three types of neural networks, optimized either for object classification, numerosity, or both. Using a novel dataset in which both training and evaluation images include daily-life objects, we analyze layer and single-unit selectivity on a range of conditions, varying the visual properties of our evaluation images. Our results suggest that numerosity classification performance is exclusive to numerosity trained networks. Moreover, we observe a discrepancy between single-unit numerosity selectivity, compared to overall network performance. This suggests that numerosity may be represented through different encoding patterns than previously assumed.