They cost me a lot of money.

It’s a pretty weird thing to collect.

But they help me every single day.

They are a bunch of historical documents of minor historical significance—it’s probably a stretch to call them artifacts—that remind me to do one of the most important but most difficult things in the world. A thing that most of us are not very good at.

As it happens, they come from the office of President Harry S. Truman.

You see, shortly after he became president, Truman was invited to the fifth annual Roosevelt Day in Chicago.

His secretary wrote an inner-office memo to ask if they should start saying no to these sorts of requests with all the demands he had on his schedule.

“The proper answer underlined, HST”, he wrote back.

It’s this last document that I have framed and hung in my office next to two pictures of my kids. It’s the idea that the proper response is to say no because everything we say yes to is, in fact, saying no to something else. No one can be in two places at once. No one can give all their focus to more than one thing. But the power of this reality can also work for you: Every no can also be a yes, a yes to what really matters. To rebuff one opportunity means to cultivate another.

I have some other mementos from Truman’s correspondence, too. One comes from 1969 when Truman was 85 years old, still working, and world famous. Sensing his time is limited, he’s explaining his long-standing policy to not answer questions from every random person who contacts him.

My editor gifted me this memento as a reminder to stay on task. Writing demands that we be a little selfish, it requires that we tune out and tune in, otherwise we can’t do what we do. But I think it’s important to realize that Truman wasn’t some selfish jerk. First, he was polite in the way he (or his secretary) declined. And second, as this other document I have demonstrated, he found a way to scale his efforts.

Truman was realizing that his book(s) were a way to help the most people most effectively. He knew that to be of service to one person at the expense of a large audience was a mistake and that it wasn’t rude to say no to one person if it meant he could say yes to more.

Anyway, I have all these reminders because I need them.

I once heard someone say that early in our careers, we say yes to everything so at one point we can afford to say no. As I’ve been lucky enough to succeed as a writer, I’ve seen my inbound requests go way up. The Daily Stoic email alone goes out to 750,000 people every day. It gets hundreds and hundreds of replies (not all of them nice!). I get requests for speeches and advice and consulting opportunities and to be on podcasts. I get invites to dinner. I have friends who want to do things. Fans come into my bookstore wanting to talk. Neighbors have things they need. Trolls try to provoke me on social media. I have my kids, my wife, and my own interests.

It is literally impossible to do it all. Especially if I want to do any of them at a high level. As I wrote a little while back, I’ve passed on everything from trips to the Super Bowl, a vacation on Necker Island, and more than a few different ghostwriting opportunities. I’ve turned down large amounts of money to do books for people, to travel and speak at events. I turned down a job in a presidential administration (although, as I wrote at the time, there were multiple reasons for that).

A younger me would have thought these things crazy to pass up on. In fact, I used to have trouble saying no to far less crazy things. I used to get Daily Stoic email replies directly to me. I used to post my email address on my website, and I would respond to everyone and everything. I would reply when people tagged or messaged me on social media. And I loved doing it—years after I replied to a reader’s email, they’d follow up with how my reply helped them. That felt great, but I’ve had to realize that saying yes—whether to an all-expenses-paid trip to a remote island or even to a short email inquiry—takes away from something.

A few years ago, Dr. Jonathan Fader, an elite sports psychologist who spent nearly a decade with the New York Mets, gave me a picture of Oliver Sacks in his office. Behind Sacks, who is speaking on the phone, is a large sign that just says, “NO!” I have that photo hanging on the wall in my office, just a few feet from those Truman memos. Above it there is a photo of my youngest son, and below it, a photo of my oldest. I can see them—all three photos—out of the corner of my eye even as I am writing this.

It’s my constant reminder that everything I say yes to is taking me away from the two people I’ve already promised so much of my time to. And from my writing, which is not only the thing that is most meaningful to me and how I make my actual living, but it’s how I can help the most people. So if I say yes to one random person, I’m saying no to a lot more people by taking that time and energy away from my writing.

In tech, they speak of “feature creep”—when a founder or a project manager isn’t disciplined enough to protect the core concept of an idea and allows too much to be jammed in it. Trying to please everyone, they end up pleasing no one. To try to do everything is to ensure you’ll achieve nothing.

This weaker part of ourselves that cannot say no to requests for our time, the part that tries to go along with everyone, perhaps deep down wants that same excuse—if we agree to their thing, then we don’t have to answer for the poor performance of our thing when it’s time for a full accounting. It allows us to say, “Well, if only I weren’t so busy . . .”

The self-disciplined part of us, on the other hand, says, like the Queen’s motto, “Better not.” Or maybe we borrow the quip from E. B. White, when asked to join some prestigious commission: “I must decline,” he said, “for secret reasons.” A clerk of Sandra Day O’Connor once said with reverence, “Sandra is the only woman I know who doesn’t say sorry. Women would say, ‘Sorry. I can’t do that.’ She would just say, ‘No.’”

No, Sandra liked to point out, is a complete sentence.

Say no. Own it. Be polite when you can, but own it.

Don’t say maybe. Don’t give a bunch of reasons (which invite an argument). Don’t push it until later.

Say NO.

Help the people who work for and with you to do the same. Jony Ive, the designer at Apple, once recounted that Steve Jobs was always asking Ive and other Apple employees what they were focused on and specifically, “How many things have you said no to?” because to focus on one thing requires not focusing on other, less important things. A device can’t have every feature.

Life is about tradeoffs.

And this is your life. And saying no is your power.

By seizing it, you become powerful. More powerful in fact, than some of the most powerful people in the world who happen to be slaves to their calendars and ambitions and appetites. The conquerors who rule over enormous empires but are slaves to solicitations. The billionaires who fear missing out. The leaders always chasing the shiny new thing. Who cares if you have achieved extraordinary things when you are punished for it by having even less freedom day to day?

It feels like you’re free because you’re choosing, but if the answer is always yes, that’s not much of a choice.