Does it really have anything to do with my neighbour, its my wall?

Yes it does if it falls into the specifics as details in the Party Wall Act etc 1996, as Party Wall are “jointly Owned” any interference or works that could compromise the Party Wall has to have both sides agreeing to any works that are to take place.