Well, there is no general agreement regarding how to use the addr: tags I'm afraid. The main addresses wiki page [1] says that address informations should be unique (one addr:housenumber per street). Obviously, a building may host more than one POI, sharing the same address. Thus, tagging POI with addr: doesn't sound like a good idea, hence the contact:addr: tags. Think a common practice is to have a dedicated point for the addresses (tagged with addr:).

[1] osm.wiki/Addresses