The concerns come as Danbury has become a hotspot for arrests by federal immigration authorities.