Most government offices were in easy walking distance of the White House. The only ones which the President regularly visited were the War Department and the War Department’s annex — the Winder Building. These buildings were in a military complex to the immediate west of the White House. Representatives of other departments were more likely to come to the White House to see the President — or to run into him on the White House grounds. Those included those cabinet members headquartered in buildings directly to the east of the White House — the small State Department Building and the large Treasury Building, where the Attorney General had his small quarters.
Mr. Lincoln’s visits to other buildings — like the Post Office or Patent Office — were sometimes ceremonial. Many government buildings — including the Capitol — were pressed into service in 1861 as dormitories for Union troops. On other occasions, he visited the Smithsonian and the Capitol for public lectures. More often, he visited the Capitol for practical reasons — like signing legislation. His annual messages to Congress had streaks of rhetorical brilliance but they lacked rhetorical flourishes. [These] “messages” were delivered by secretaries to be read by an official in Mr. Lincoln’s absence. Mr. Lincoln’s most notable visits to Capitol Hill on the occasion of his First and Second Inaugurations. On both occasions, he gave memorable addresses before crowds anxious to learn the direction his administration would take — first with war and second with peace.
Since the President’s most vocal opponents — on both the Republican and Democratic side — occupied the Capitol end of Pennsylvania Avenue, his relations were often difficult. But most often, relations were conducted at the White House, not at Capitol Hill. When on December 17-18, 1862, Republican Senators sought to oust William Seward from the Cabinet, they went to the White House on both days.
Cabinet members whom the President did not regularly visit saw the President primarily at Tuesday and Friday cabinet meetings. Indeed, Secretary of the Treasury Salmon P. Chase and Attorney General Edward Bates pushed for such scheduled meetings in order to counter-balance the President’s frequent interactions with Secretary of State Seward. The President’s interactions with Secretary of War Edwin Stanton later overshadowed even those with Seward. Understandably, therefore, they saw less need to attend Cabinet meetings.