Note: groups.io will be down for maintenance this evening, starting at 11pm Pacific Time (6am Tuesday 8/3/2020 UTC), for up to two hours.
You can't get rid of some "group by" in contacts, but I think I know what's happening here. The default "Group by" is company, and most of my contacts have none.
Try ALT+V,E,E, and if that doesn't do it, then ALT+V,E,C. I am presuming that the company grouping is currently collapsed, which means each contact's company will be shown on a per-item basis. When it's expanded, it disappears (which is counterintuitive).
But those two commands either expand or collapse the current group view in contacts. So give it a try.
Brian - Windows 10 Pro, 64-Bit, Version 1909, Build 18363
Life is the art of drawing sufficient conclusions from insufficient premises.
~ Samuel Butler, 1835-1902