I’ve spent my whole life at the intersections of ethnicity and identity and sexuality. Raised by my Ashkenazi Jewish mother without my black Baptist father—and with, perhaps, the most “Jewish”-sounding name imaginable—I’m far too accustomed to people prying into my racial background and family structure.

Well into adulthood folks would openly wonder “how I knew” the woman, my mom, sitting or standing or chatting next to me. And while I understood that my own family—what, with its two dads—would also invite intrusion and confusion, I hoped (if not prayed) that folks would never, ever question my inviolable status as their father.

So far, most haven’t—not really—but I know it’s merely a matter of time. In Manhattan, where we live, there’s nothing unusual about dark-skinned women toting white kids around town; they’re usually the nanny. Indeed, generations of white American children have been raised by black and brown women whose servitude—often forced and unpaid—kept them from their own families and children as they toiled away in the “big house.”

On the other extreme, there’s been a mini “boom” in white female celebs—think Sandra Bullock, Angelina Jolie, and Madonna—adopting African and African American children. Prominently featured in endless paparazzi shots, the resulting families have helped accustom many to the optics of the white-mom/black-child trope.

But few parallel examples exist for the opposite setup: Men like myself, dark-skinned with light children. Which is why I’m so often afraid.

America was built on the fear, loathing and labor of black men; we are the literal bogey men—our black lives often truly do not matter. At the root of this legacy is black male access to white privilege, property, and people—particularly white women and children. America’s ghastly love of lynching was steeped in fears of miscegenation while the pernicious “one drop” rule —which declared that any amount of African American blood rendered folks black—ensured that American families would maintain apartheid-like levels of segregation no matter their actual skin color.

While black women were “permitted” to raise white charges, social, cultural and institutional constructs did everything possible to keep black men from having any legitimate claim to white children. There was, literally, no real reason for intimate relationships between the two.

But where does that leave families like my own? I’m not entirely sure. For one thing, we are certainly not alone: Since 1970, the percentage of mixed-race children has spiked from 1% to 10% today, according to the Pew Research Center. And yet there are clear indications that the US isn’t moving forward in the fight against racism, but backwards.

In my case, my boys are still too young for us to attract much notice—though I see people eyeing us in confusion almost every time we’re in public. I worry when they’re older and louder and—like most boys—fussier and disobedient. I worry…say…about the day on the subway when one—though likely both—refuses to sit in their seatsproperly or hold on to a safety rail. I fear the resulting discipline—direct, stern, and loving—might catch the eye of some well-meaning white person who could challenge my parentage, question my legitimacy and—entirely baffled—call the police. They cops might ask me to “prove” my parentage, like the white woman with a biracial son who was asked to confirm she was his mother as she tried to board a Southwest Airlines flight last month. Or worse.

I wish I lived in a world where this were mere hyperbole; I wish such fears were far-fetched and unfounded,  But unlike my dreams of becoming a father, these wishes will probably never come true.

In the meantime, much like my mother before me, I go about the quotidian duties of being a parent—too sleep-deprived and diaper-laden, too absorbed in my sons’ sheer deliciousness—to allow myself to fully live in fear.

There have been moments—mostly benign, but occasionally cringe-worthy when our sense of normalcy has been disrupted. Last summer, when the boys were just infants, I was sitting in an ice cream shop slurping a scoop which Luca was eyeing greedily. The woman next to me—white and middle-aged and completely unremarkable—couldn’t take her eyes off of us. She seemed sad for my boy—all covetous and gelato-deprived. But he was just too young for a taste. Still, she clearly needed to get a word in, but was obviously too confounded by our relationship to know exactly what to say.

So she chose her words carefully, slowly and diplomatic. “Perhaps this nice man you’re with will buy you some ice cream,” she said looking Luca’s way and clearly baffled by my presence. “Of course I will,” I retorted, “but not until he’s older and I think he’s ready, because I’m his father.”

This article is part of Quartz Ideas, our home for bold arguments and big thinkers. 

📬 Sign up for the Daily Brief

Our free, fast, and fun briefing on the global economy, delivered every weekday morning.