Dolley P. Madison - Dolley Madison's World

Dolley Madison's world was small, geographically -- she inhabited only a few places in her lifetime -- but her influence has loomed large in almost every place she dwelt. She was born in New Garden, North Carolina, but her parents moved back to their family home in Virginia when she was still an infant. Her childhood was experienced entirely in rural Hanover County, Virginia. From there, Dolley's father moved his family to Philadelphia when she was fifteen. It was in this sophisticated city that Dolley grew to womanhood, married her first husband, became widowed, and met her second husband, James Madison. They would wed at her sister's home, Harewood, in what is now West Virginia, then move back to Philadelphia for the first three years of their marriage. When the nation's capital moved to Washington City, one of the nation's leading political couples, James and Dolley Madison, would soon follow. It was in that bustling and burgeoning setting that Dolley Madison came into her own as the nation's first First Lady. She would later return to the city as a widow after James Madison's death. In the years between his presidency and his death, however, the Madisons would live out their retirement years at his family estate, Montpelier, in Orange County, Virginia. These few places -- New Garden, North Carolina; Hanover County and Orange County, Virginia; Philadelphia; Harewood, in what is now West Virginia; and Washington City -- comprise the key sites of Dolley Madison's world.

Dolley's parents, John and Mary Payne, met and married in Hanover County, Virginia, just north of Richmond. They soon moved to North Carolina, however, where their first daughter was born. After only a few years, and before Dolley had turned one year old, the family returned to Hanover. The reasons for their return are unclear, but it is certain that life in central Virginia would mean having more family members nearby. Family tradition has it that the Paynes lived at the estate known as Scotchtown when they arrived, then later sold the home to Patrick Henry. Dolley's niece reports that Dolley remembered "the mantelpieces, they were of black marble supported by white figures, there were twenty rooms on a floor, every one had marble hearths and mantels." Scotchtown underwent extensive remodeling in the nineteenth century, and if it ever had the black marble hearths and mantles that Dolley reportedly described, the originals are gone now. Nonetheless, it is clear that the house never had more than eight very large rooms on the first floor and an extensive but undivided attic above.

In fact, public records refute much of the official family lore, since it would appear incontrovertible that John Payne never owned this stately home. In 1766, the owner of Scotchtown died, and the executors of the estate put the property on the market. The home wasn't sold until 1770, and Patrick Henry paid the money for Scotchtown in 1771-72, directly to the executors of the estate. Therefore, it seems impossible that John Payne could have owned the estate in the intervening years. It is possible (though no proof exists) that the Payne family leased the house from the time they returned from North Carolina in 1769 until Henry paid for it in 1771. Henry lived at Scotchtown only until 1776, when he became governor of Virginia, then sold the plantation in 1778.

Nonetheless, even if the Payne family never owned the house, it would not be surprising if they were frequent, and perhaps even extended, visitors there. Dolley's mother and Patrick Henry were cousins, and family relations mean a lot in Virginia. Henry first made payments on Scotchtown in 1771, and that was the same year that John Payne purchased 176 acres of Coles Hill (an estate which belonged to Dolley's maternal grandfather). In all likelihood, a house would have had to be built on this newly purchased portion of the Coles Hill property before the Payne family could move there. Coles Hill is about 10 miles from Scotchtown, and the Paynes may have stayed at the Henry residence while their home was being built. The Payne's home at Coles Hill is described as being a modest frame house with only two rooms on the main floor. The property where the Coles Hill plantation once stood is now completely wooded over, and there are no known structures standing on it.

Coles Hill was about six miles from the Cedar Creek Meeting House, the focal point for Quakers in this part of the County. There, John and Mary Payne both served as clerks and elders of the local congregation. According to some accounts, Dolley received at least part of her education at Cedar Creek as well (which would have meant a six-mile trek each way to school). Whether or not that account is true, the Paynes were on the subscribers' list for the school there. Accounts about her early schooling are few and unreliable, but she surely received at least a rudimentary education somewhere -- even if it was at home -- and at some point even acquired a little French. Even if Dolley never went to school at the Cedar Creek Meeting House, it certainly would have been an important center in the lives of this devout Quaker family until they moved away in 1783.

Dolley spent her formative years -- from 1783 to 1797 -- in the city of Philadelphia. During this time, the city was notable for three things. First, it was the most cosmopolitan city in America; second, it became the nation's capital during the 1790s; and third, it was the hub of Quakerism in America. John Payne, Dolley's father, moved his family here for the third reason alone, but the first two features of the city would have a more lasting influence on his daughter Dolley.

This 1776 map of Philadelphia shows the city largely as it would have appeared to the fifteen-year-old girl who burst onto the scene only a few years later. It is numbered to show the places that would have been meaningful to her during her time in the city.

Since most of the street names in the historic district of Philadelphia have remained the same, this map can be used to take a Dolley Madison walking tour of the city. To download a PDF file of the map, click here. To download a key to the map in Word format, click here. Below are listed the Dolley Madison highlights of Philadelphia, in chronological order.

1. Drinkers' Home: When the Payne family moved to Philadelphia in 1783, they stayed first with some friends who were also fellow Friends (in other words, Quakers): Henry and Elizabeth Drinker. They lived on the northwest corner of Front Street and Drinker's Alley. Elizabeth Drinker is noteworthy in her own right for having left to posterity a diary full of local doings that spans many years. Her notes on the Payne family -- though scant -- offer some of the very few records we have of Dolley's early life. Nothing remains of the original Drinker house, but nearby Elfreth's Alley is remarkably well preserved, and it is worth a saunter down this lane to ponder what Miss Dolley Payne's neighborhood looked like when the teenager first arrived in Philadelphia.

2. Northern District Meeting House, also known as the Bank Meeting House, was already more than 100 years old when the Paynes moved to Philadelphia in 1783. The Paynes had a certificate of transfer from the Cedar Creek Meeting House, in Hanover County, Virginia, directly to their new spiritual home. They were members until 1786, when they transferred to the Pine Street Monthly Meeting House. Within a few years after the Paynes left, the Northern District Meeting House relocated as well. Today, there is nothing of the old structure to be seen.

3. Paynes' Home and Starch Business: John Payne had been a planter in Virginia, and he owned a store in North Carolina. After moving his family to Philadelphia, however, he decided to try his hand as a starch manufacturer. Unfortunately for him, Philadelphia experienced an economic downturn in the 1780's, and laundry starch was a luxury item that many families decided they could forgo. His business had failed completely by 1789, and John Payne was read out of the Pine Street Meeting House for failure to pay his debts. Despondent, Dolley's father took to his bed and turned his back on the world for much of the remaining three years of his life. The business was at 89 Elm St., which is now 231 New St. Presumably, the Payne family lived on the upper floors above the business. The structure there today appears to be a garage, and is just a stump of its former self.

4. Franklin Court: According to Dolley's niece, Mary Cutts, Miss Dolley Payne was a friend of Benjamin Franklin's daughter, Mrs. Sally Franklin Bache, who was 25 years her senior. She was therefore among the young people who visited the lively but elderly Franklin in his last days. She even received a rebuke from him after an episode that may be described as the only instance ever recorded of Dolley's rudeness. Apparently, the young lady had slapped the hand of one of her friends, whereupon Dr. Franklin admonished her, "Dolley, Dolley I am ashamed of thee!" The rebuke was a "lesson to her through life," reported her niece.

5. Pine Street Monthly Meeting House: The Payne family transferred here in 1786 from the Northern District Meeting House. John Payne quickly became an elder and lay preacher (at least, until he was read out of the Society in 1789). Here, Dolley Payne met the eligible bachelor, John Todd. Later the two would wed in this house of worship on January 7, 1790. Nothing remains of the original structure today.

6. Todds' first home: Shortly after John Todd married Dolley Payne, he hung up his shingle as a practicing lawyer at 85 Chestnut St. (which is approximately 247 Chestnut today). It is presumed that this is the first place that the newlyweds lived after the wedding. By some accounts, the Todds lived with the Payne family immediately after the wedding, which, if true, would mean that this address was a transitional home between the John Payne starch business on Elm St. and the Mary Payne Boarding House on N. Third St. But that detail is not known for certain.

7. Payne Boarding House: After her father's business failed, Dolley's mother, Mary Payne, opened a boarding house in 1791 which served many of the politicians and other important personages who were flocking to the nation's temporary capital. Her business venture would eventually improve the marital prospects for two of her daughters. One of her boarders was George Steptoe Washington (a favorite nephew of the president), and he would elope with Dolley's younger sister, Lucy. And Senator Aaron Burr also stayed here. Burr became so well acquainted with (and respected by) Dolley that she named him in her will to be guardian of her only child in the event that Payne Todd should be orphaned. Burr's close acquaintance with Dolley Todd is what prompted James Madison, an old college acquaintance, to solicit him for an introduction to the widow. The boarding house was at 96 N. Third St., which was later renumbered to 150 N. Third St. The structure there today is, like the old starch business, an unprepossessing garage.

8. Todd House: About a year after their marriage, the Todds were able to buy a house of their own on the corner of Fourth St. and Walnut. There, John Todd practiced law on the first floor, in the room facing Walnut. The parlor and the family's bedrooms were on the second floor, and Todd's assistants lived on the upper floors. After Todd's death in 1793, the young widow attracted many potential suitors. In the spring of 1794, Dolley wrote to her friend, Eliza, "Dear friend, thou must come to me. Aaron Burr says that the 'great little Madison' has asked to be brought to see me this evening."

The two entrances face Fourth St. The left entrance leads to a private kitchen. Right, the upstairs parlor where Dolley would presumably have entertained.

Before marrying Madison, Dolley conveyed this property in trust for her two-year-old son from her first marriage, John Payne Todd. Today, the restored house is owned by the National Parks Service and is open for tours.

9. Free Quaker Meeting House: When John Payne, Dolley's father, was expelled from the Pine Street Meeting House, he joined the Free Quakers (also called the "Fighting Quakers," because they had supported the Revolutionary War). They held their meetings at Fifth and Arch St. There he became an active member in good standing, but his family did not follow him into his latest conversion. Today, the Meeting House has been restored, and historical interpreters help bring the place to life for visitors.

10. Free Quaker Burial Ground: When Dolley's father died in 1792, he had his funeral service in the Free Quaker Meeting House, then was buried in the Free Quaker Burial Ground a few blocks south, on Fifth Street, north of Spruce. The burial ground is no longer there; in 1905, the remains were moved to various other sites.

11. Grays Ferry: When Dolley was a carefree young maid, she was known to have "had a frolick" with friends at Grays Ferry, which was then a resort town about three miles southwest of the city of Philadelphia (today, it is a neighborhood of Philadelphia). Her mother, worried that such behavior was not proper for Quakers, consulted her friend, Elizabeth Drinker. Together, they decided that such outings would be forbidden to the young people in the future. Later in Dolley's life, however, the place would have much more painful associations. When the yellow fever epidemic broke out in late summer of 1793, John Todd removed his wife and their young family to Grays Ferry. Dolley survived the epidemic, but her husband and their newborn son perished on the same day. Much later, when Dolley Madison returned to Philadelphia in 1805 to treat a knee ailment, she stayed in "Greys Green House" in Grays Ferry.

12. President's House: Shortly after James Madison began courting Dolley Todd, Dolley was summoned to the President's House by none other than Martha Washington. Mrs. Washington asked if the rumors were true that she was engaged to James Madison. Flustered by the unexpected question, Mrs. Todd answered no. "'If it is so,' Mrs. Washington continued, 'do not be ashamed to confess it: rather be proud; he will make thee a good husband, and all the better for being so much older. We both approve of it.'" Dolley was related to the Washingtons through the marriage of her younger sister, Lucy. Nonetheless, as a pious Quaker woman, Dolley had never before attended the president's levees. But after her marriage to Madison she would regularly attend these and other festivities. Today, only the foundation of the President's House -- recently unearthed by a team of archaeologists -- can be seen.

13. Madisons' First Home in Philadelphia: When James Madison began courting Mrs. Dolley Payne Todd in the spring of 1794, he was living with James Monroe at 4 North Eighth St. Early that summer, Monroe sailed to Paris after being named Minister to France. James and Dolley spent that summer (separately) visiting family in Virginia. Then they reunited at Harewood -- her sister Lucy Washington's home in what is today West Virginia -- to get married on September 15, 1794. When the newlyweds returned from their honeymoon later that fall, they set up housekeeping in the same residence. That site is currently the location of the old Strawbridge's department store -- an impressive building in its own right -- but unconnected with anything from the Madisons' era.

14. Congress Hall: At the time that Dolley married her second husband, James Madison was one of the most prominent members of Congress in the nation's temporary capital of Philadelphia. Although there are no records of Mrs. Madison visiting Congress Hall while she lived here, we do know that she used to frequent such debates in Washington City. (She temporarily suspended her visits during the War of 1812, because the vitriol directed at the president during these debates made her too angry.)

15. Madisons' Last Home in Philadelphia: During Madison's last sessions of Congress, in 1795-97, the couple lived at what is now 429 Spruce Street. In an example of eighteenth-century high-pressure salesmanship, Madison received a note in June of 1795 advising him to accept the terms at this residence as soon as possible: "I really do not know how you can do better than to secure this House at a time when Rents are every where Rapidly advancing and Houses in general so scarce as to be very difficult to be procured . . . You will therefore be good enough to let me know your determination early as may be." From this residence the couple moved to Montpelier in 1797 and spent the next four years "in retirement," before moving to Washington in 1801. The building still stands, and there is a plaque on the wall, but it is a private residence not open to visitors.

16. Dr. Physick House: As a child, Dr. Philip Sing Physick had been a student of John Todd, Sr., and had boarded with John Todd, Jr., his childhood companion, who would grow up to be Dolley's first husband. Therefore, it is not unreasonable to suppose that the Todds were frequent visitors to this home while they were living less than two blocks away on Walnut St. In 1805 Dolley Madison, then the wife of the Secretary of State, was suffering from a painful knee ailment. Travelling to Philadelphia from Washington City, she sought treatment from the eminent Dr. Physick. Although her healing took longer than he had predicted, it went well. And the good doctor had excellent bedside manners as well. He buoyed the spirits of his patient by talking to her of her husband (from whom she had never before been separated). Dolley wrote to James that Dr. Physick "regards you more than any man he ever knew and nothing could please him so much as passing his life near you." She was clearly impressed with the doctor's abilities as well. A few years later she recommended his talents to her sister, and she also consulted him about her eyes shortly after James Madison died in 1836. On that occasion, Dr. Physick recalled, "I remember with pleasure, that so long ago as the year 1805 Mrs. Madison when in Phila did me the honor to select me as her medical adviser, & so frequently since has favored me with friendly reccollections as to excite in me feelings of the most grateful remembrance." Today the house is restored and open for tours.

Back to top

James Madison was visiting his family at Montpelier in the summer of 1794 when he got word from the widow Dolley Todd, whom he had been courting for only a few months, that she consented to be his bride. She was spending that summer traveling in other parts of Virginia to visit her own family. The two met at Harewood, her sister's home, for a quiet family wedding on September 15, 1794. Two years before, Dolley's sister, Lucy Payne, had eloped with George Washington's nephew, George Steptoe Washington. Harewood has remained in private hands ever since, passed on through the direct descendants of George Steptoe and Lucy Washington. This private residence is in Charles Town, near Harper's Ferry, in what is now West Virginia.


The Exterior of Harewood today, very little changed from when the Madisons were wed there on September 15, 1794.	The room where the Madisons were married. It has remained untouched since that day.

It is difficult to evoke the Washington City that Dolley Madison would have seen if one visits the bustling metropolis of Washington, D.C. today. The city had been the nation’s capital for less than two years when the Madisons arrived in 1801. The swampy land had not yet been adequately drained, and farms and pastures dominated the landscape. Many people, including Dolley, noted the unhealthy conditions prevailing as a result of that marshland. One visitor observed, it was “a city of streets without buildings and buildings without streets." Frequently, those streets were no better than a muddy morass, strewn with rocks and tree stumps. Another visitor remarked that “neither churches, theatres, nor colleges, could I discover so lately as the summer of 1806.” The newly built White House and Capitol Building towered over any other nearby structure.

Dolley Madison would live approximately 25 of the next 49 years in the burgeoning Washington City, making her presence felt at, if not actively making her mark on, numerous historical sites. Glimpses of what Dolley saw then can still be seen today. Below is an 1836 map of Washington City (right-click here to download it as a JPEG file).

B) The Capitol Building: The site of President Madison’s inauguration in 1809. Many ladies, including Mrs. Madison, frequently sat in the galleries of the Supreme Court (which was located in the Senate wing) or the House, listening to the legal and political debates. But during the rancorous period surrounding the War of 1812, Dolley found the vitriolic speeches in the House more than she could bear. She wrote to a friend on January 14, 1813, “I stay quietly at home – as quietly as one can be, who has so much to feel at the expression, for & against their conduct.” Clearly, Congress had made their peace with the venerable Mrs. Madison by January 8, 1844, when they bestowed on her the singular honor of a seat on the House floor whenever she chose to visit. Tours of the Capitol can be arranged through one’s congressman or visit www.visitthecapitol.gov.

C) Long’s Hotel: Immediately following the inauguration, the Madisons received formal visits at their home on F St. That evening, Long’s Hotel hosted the nation’s first Inaugural Ball. 400 guests were invited, but it was said that anyone could attend who could afford the ticket price. Contemporary accounts described the ball as “the most brilliant and crowded ever known in Washington.” Dolley’s friend, Margaret Bayard Smith, recalled that Dolley “looked a queen” in her buff-colored velvet gown and turban. Currently this is the site of the Library of Congress.

An 1816 watercolor by Benjamin Latrobe,
showing a gutted White House.

D) The President’s House: President Madison and the Lady Presidentress moved into the President’s House immediately after his inauguration in 1809. The “President’s Palace” was already informally known as the White House by then. Even before she had completed the transformative interior decorating blitz with her collaborator, Benjamin Latrobe, Dolley had launched the tradition of lavish entertaining in the White House. She arranged her crowded Wednesday night Drawing Rooms (called “squeezes”) with no less care the than the formal balls and state dinners they hosted. The couple would remain here until the British burned the edifice in 1814. Requests for tours must be made through one’s Congressman, and submitted at least three weeks in advance. See www.whitehouse.gov/history/tours.

Belle Vue, today known as
Dunbarton House

E) Belle Vue: Dolley Madison was whisked to the home of Charles Carroll on the afternoon of August 24, 1814, when the British marched into Washington. The core of the house is original, but was moved slightly from its original location when the Georgetown and Washington segments of Q street were united in 1915. It is currently called the Dumbarton House, owned by The Colonial Dames, and open for viewing: 2715 Q St., NW, Georgetown, (202) 337-2288.

The Octagon House, today owned by the American Architectural Foundation

F) The Octagon House: Following the burning of the White House, and a month’s stay at their old residence on F Street (then the home of Anna and Richard Cutts), the Octagon House became the new home for the President and his wife. Dolley immediately resumed the tradition of the Wednesday Night Drawing Room, her social graces helping to soothe an irritable public weary of war. These parties were not mere frivolous entertainments; they had a serious purpose. There was an earnest debate at this time about moving the capital to Philadelphia, and it was incumbent upon the Madisons to convince the skeptics that Washington City was resilient and deserved a second chance. The President’s office was on the second floor, and the Treaty of Ghent – ending the war of 1812 – was signed there on February 17, 1815. Later that year, the Madisons would move to the Seven Buildings at 1901 Pennsylvania Ave, at the corner of 19^th St. (nothing remains of the Seven Buildings). They would remain here until they retired to Montpelier in 1817. The Octagon House is currently owned by the American Architectural Foundation, which offers limited visiting hours to the public: 1799 New York Ave, (202) 638-3221, nps.gov/history/nr/travel/wash/dc22.htm.

G) The Washington City Orphan Asylum: After the war of 1812, Dolley became the first First Lady to champion a charitable cause: an orphanage. She donated $20 and a cow, and helped to cut patterns for the girls’ clothes. The site is roughly between 900 and 906 H Street, but the building is no longer there.

The "Dolley Madison House," where
she lived during her widowhood.

H) Home of the widowed Mrs. Madison: After her husband’s death, Dolley returned to Washington, living at the former home of Anna and Richard Cutts, her sister and brother-in-law. There she remained for most of the years between 1837 and her death in 1849. She immediately resumed the active social life which made her such a beloved fixture in Washington City. The first person to pay his respects to her was President John Quincy Adams. In May of 1848, the house caught fire, but Dolley refused to leave before her husband’s papers were secured. Though called “The Dolley Madison House” today, this bright yellow building is currently being used as office space for the U. S. Court of Appeals, 717 Madison Place. It is not open for public viewing, though there is a historical marker on the wall.

St. John's Church, as seen in this 1816 watercolor by Benjamin Latrobe.

I) St. John’s Episcopal Church: Dolley Madison was confirmed here on July 15, 1845. The next day she wrote to her nephew: “And now my dear Richard I will tell you on what our thoughts have dwelt a great deal – it was to become worthy of membership in the Church which I have attended for the last forty years.” Her funeral would be held at this church exactly four years after that letter was written, and hundreds would come to pay their respects. The City of Washington closed down to provide a state funeral for her, the largest the capital had ever seen. President Taylor referred to her as the “First Lady” in his eulogy, the first time that term was used. The church is located at the corner of 16^th and H Streets, facing Lafayette Square.

J) The Washington Monument: Dolley Madison, along with the widows of Alexander Hamilton and John Quincy Adams, united forces to raise funds for the Washington Monument. Mrs. Madison was of course invited to be present at the laying of the cornerstone on the Fourth of July, 1848. She sat at the dais with President Polk and numerous other distinguished guests, including a young Congressman named Abraham Lincoln. It was to be her last public appearance.

       When Dolley Payne Todd married James Madison in 1794, she united herself to a leading family of landed gentry in Orange, Virginia. And she married the eldest son in that family. But it would be a few years before she even saw the estate she would eventually inherit, and more than two decades before the Madisons relinquished political duties entirely and settled permanently into the idyllic life (and precarious livelihood) of Virginia planters. When the Federalists took control of the federal government in 1797, James Madison retired from Congress and took his bride to live at Montpelier. For the first time in his life, he embarked on the full-time management of the estate, estimated at about 10,000 acres. To all appearances, it may have been intended as a permanent move.

          Madison's parents were still alive when the young couple moved in. And finding his childhood home too small to accommodate his new family (which included Dolley, her young son, and her sister, Anna), he initiated an ambitious remodeling project. He extended the north side of the house, which considerably enlarged Montpelier and made it into a duplex.  The family of James Madison Jr. lived in the northern part of the house, and the older generation remained in the original southern half.

          Two important events occurred in the Madisons' lives in 1801: James Madison Sr. died, and James Madison Jr. was named by President Jefferson as secretary of state. Jefferson would have preferred that Madison take up his duties right away, but he stayed behind at Montpelier for some months in order to settle the many tangled details of his father's estate before he left. Even though Madison was now the legal owner of Montpelier, he would spend most of his time over the next sixteen years in Washington: first as secretary of state, then as President. The Madisons during these years primarily used Montpelier as a summer retreat from the oppressive heat of the nation's capital.

          During Madison's first term as President (1709-13), the couple once again chose to refurbish and enlarge Montpelier. Interior walls were knocked down in order to transform the duplex once again into a cohesive home, and wings were added on each side. Each wing was graced with a terrace on top and a kitchen down below.