Not to mention most of the sunlight we receive is infrared, and our atmosphere does block some chunks of it, and those longer infrared bands really isn't particularly useful for solar power, and to a lesser extent UV radiation as well.
You need that infrared in space anyway to toss it away/radiate/induce/create cooling as discussed earlier in the thread, while as you say on Earth it's useless/detrimental/causes heat.