Maybe... he lost his clothes (I don't know how that would happen, but this is Jack) and he's borrowing the t-shirt and khakis? If I had to measure a guess at who's clothes they were, they look like Rhys' clothes, but I don't think they're the same size. At all. Though we've seen Ianto's casual clothes once and probably not enough to judge what his casual wardrobe is like so maybe they're his (I say this just because I'd guess them to be closer in size, not for any Jack/Ianto-y reasons).

I'm just trying to think up a good reason for him wearing random clothes that could have been product of his nudity.