How worried should you be about an elderly relative at the wheel?

Many Americans (40%) would rather talk to their elderly parents about funeral arrangements or selling their home than ask them to curtail their driving, new research finds. Yet only 16% of Americans think elderly relatives should make that decision themselves, the survey of 1,000 adults by Caring.com, a site for people caring for aging family members, found. Relatives of elderly drivers are split about who should take responsibility. Adult children would rather a doctor/caretaker (29%), family member (25%) or the Department of Motor Vehicles/the government (23%) take responsibility.

Whether an elderly relative should still drive is a taboo subject. “Senior driving is one of the most challenging topics for caregivers,” says Andy Cohen, chief executive of Caring.com. “Driving is all about independence and one of the most emotional things for anyone to give up.” On a practical level, he says it’s hard to get around most parts of the U.S. without a car, but improvements in technology with Google’s GOOG, -1.97% self-driving cars, which have a maximum speed of 25 miles an hour, will enable many seniors to drive longer. “That will be great,” he says. “It will help make driving safer.”

Read: You killed me at hello: 26% of car wrecks involve phones

That said, senior drivers — aged 65 and above — are among the least dangerous and reckless drivers on the road, safety experts say. “It’s really easy to glance at data and jump to the conclusion that older drivers create more crashes,” says Jake Nelson, director of traffic safety research and advocacy at the AAA. “They’re not. If they are a hazard to anyone, it’s to themselves. They’re over-represented in fatal crashes because of their fragility. They are less able to withstand the forces of a crash that you or I walk away from and less able to recover from their injuries.”

Younger drivers are killed in more crashes, and alcohol-related fatalities peak for those aged 21 to 34. There were 2,692 people aged 65 to 74 killed in motor vehicle traffic crashes in 2012 (11 per 100,000 of the population), the last year for which data are available, according to the government’s National Highway Traffic Safety Administration, but that rose to 2,868 or 15 per 100,000 for those over the age of 74. However, 21 to 24 year-olds made up the highest percentage killed in motor vehicle crashes (19 per 100,000 or 3,436 people) while 25 to 34 year-olds had the highest number of fatalities (5,902 people or 14 per 100,000).

Read: 10 riskiest (and safest) states for drivers

There are warnings that an elderly driver may be having difficulty. Any cognitive decline will obviously require observation, Nelson says, but there are signs to watch for, including getting into more fender benders, an increase in traffic tickets or warnings from traffic police or getting lost on familiar routes in familiar areas. “These likely point to a medical condition that could make them candidates for driver rehabilitation therapy,” Nelson adds. The AAA has programs to help elderly drivers and states have different rules on license renewals for older drivers.

Elderly drivers appear open to discussions on the issue. Some 30% of Americans aged 65 and older would actually prefer their family determine whether or not they should still have a driver’s license, the Caring.com survey found; 26% of that age group prefer to make the decision themselves, while 21% would like their doctor or caretaker to make the decision. That said, only 10% of senior citizens think the DMV or government should rule on the issue. “Almost always there is something that can be done,” Nelson says. “In many cases comprises can be reached where elderly relatives still have their mobility, but maybe they only drive during the daytime, or on certain routes.”