Some people do call the place Bandaraya, at least the area surrounding the DBKL building, especially since the LRT station nearby is named as such. Back before the rapid development of KL prior to the 90s, most of central KL we know today haven't existed in its current form yet, so the historical core area was referred to as such at least by the old timers like my father and his peers who grew up in KL during the 60s.

Also downtown and city center is an interchangeable term, isn't it?