Fitz needs to mend his relationship with Karen and Jerry before his mends his relationship with Olivia. Kids trump girlfriend. My guess is that he goes up to Vermont to make amends with them. If they won’t come to him then he has to go to them.
Maybe this is where he’s at in this scene?
His clothes are casual so he’s not on official business.