The Philidor Gambit, not to be confused with the Philidor Countergambit from the Philidor Defense, is a challenge to the g5-pawn. You are trying to undermine Black's strong point on f4.
The most common reply from Black is to shore up g5 with the move 5...h6. He is not worried about exchanging on g5 as his g7-Bishop is defending his Rook.
By playing 5.h4 you are indicating a willingness to compromise the integrity of your King's fortress in order to put a strong attack into place against the enemy.
In many of the games White's Kingside does indeed become flimsy but his King proves agile. He can get across to the other wing without too much danger. What this option does give you is a fearsome center that can be the platform to a decisive incursion into Black's camp.