This is an interesting article by Jonathan Mosen, CEO of Workbridge, an employment service for people with a disability or health condition.
Discriminatory words matter. Even when used as a metaphor or said for a laugh, they create subconscious bias.
Ableist language doesn't have to be said with an intent to harm. Usually, no harm is meant, but harm occurs just the same.