Yes, it is a worry. But the swirling is inevitable over time. You can't wash a car, no matter how carefully, without at least very light swirls appearing over time. I don't see how a duster, lightly drawn across, and picking up dust, can be abrasive enough to make swirls as much as a drying towel. But, I have been wrong a bunch of times in my life.
Just used mine today Merlin. Removed quite a bit of dust and pollen but not all.8 don’t use much pressure. I thought you wash your car every time it snows??
Only when it snows/rains on ME! In between times? I get dusted a lot more than rained/snowed on around here. But it's one or the other; and I will not wash my car just to get dust off it. A patina of sorts accumulates after a week of sitting and driving around. And "patina dusters" do not exist. However, the danger that some fall into is that they start to use the duster across the same area, with the patina (i.e. adhered stuff like stray spots of whatever), to see if a subsequent pass will remove what they see; then if they'll do that forbidden thing, the next sin is to apply just a teensy-weensy bit of pressure and go over it again. Bad, bad, very bad. And then they say, "I hate dusters, they scratch my paint." Hah.
Rubbing anything on the car introduces the risk of swirls. That risk increases exponentially if you rub the car in circular motions regardless of what you put on it. Quick detailers and good, clean microfibers are no more of a risk than a drying towel or a wash mitt if used correctly, with little to no pressure and straight line passes. If you have to do more than that to get something off the car using a detailer, it should be washed off.
Waterless/rinseless washes are the same case but they tend to be better at breaking down crap on the car if it's slightly heavier dirt or something.
Rubbing anything on the car introduces the risk of swirls. That risk increases exponentially if you rub the car in circular motions regardless of what you put on it.