"My mind's made up about Brett Kavanaugh and it would take a dynamite accusation," said Graham