I tend to agree with you that fame is based on the sum of all cities.
I've been working on my fame in the cities individually. My fame in San d'Oria is maxed, my fame in Windurst is pretty high, but I have absolutely NO fame in Bastok (not a single quest). Nonetheless, my fame in Jeuno is one level from being maxed.
If it were true that fame was based on the main city with your lowest fame, then I should have next to no fame in Jeuno, yet mine is almost maxed.