There is nothing dirty, unclean, or wrong with how gardens look after winter. Why do you have to "clean up" your garden? Are the inlaws coming over to judge you and break up your marriage? If you are going to go out there and remove dead plant material there better be a point to it. Points that count include but are not limited to:
1) You're trying to increase seed germination and many seeds need sunlight at the soil surface so you're trying to get light to touch and warm up said soil surface. Therefore, you are removing material that is currently shading the soil surface too much. This might include cutting back plants really low, raking out meadow beds, or even using fire.
2) Removal of diseased material or material that is actually, really, smothering other plants to the point they actually, really, can't push through (rare). Our experience is that the smothering aspect is rare.
3) Tidying up visible areas that demand said tidying up for reasons, alas, not connected to ecology and ecosystem function. For example, the front yard lawn-to-meadow conversion of wilder foundation beds where "cleaning" is a good thing for neighbors to see you doing, because seeing active management might help them understand you aren't just "lazy" or don't care.
Your spring garden -- and all of the spent plant material -- are intentional and they are still sheltering insects and other fauna that don't all emerge on the first warm day like suburban mowers. Some insect and bug species won't emerge for weeks and months yet, in fact. In nature we don't see birds or squirrels or prairie dogs cleaning up meadows or woodlands. We are the only species that removes native vegetation, replaces it with unnatural arrangements of plants that may or may not be suited to the site or one another, and then treat these assemblages as closets or living rooms that need spring cleaning. We create more work for ourselves and I don't know why. And for some reason if you don't participate actively in this work, you are dubbed an outsider and even an outlaw. Cleaning up is just another term for oppressing nature -- but if you do it with a purpose, in sync with habitat and ecosystems, it can be a more strategic and lighter tool whose specific actions lead to specific ecological goals.