I'm tempted to think Mr. Madison might have added

"You thieving bastard" or something. Seeing as how they're in private; he was too much a gentleman to say that out in public.

Thanks to Kevin