It depends on the time of year. If you are removing frames before the winter months, you should be reducing the hive anyway. So, I would remove the frames and box(es) and winterize the hive. If this is your first harvest of the spring, you will need to add the frames back or replace them. If at all possible, try to remove and extract within a couple of days. You can leave the empty box on the hive so the bees have room at night, and add back the frames after extraction. The bees will fill them up quicker if the comb is built out and they still have some honey left over from the extraction. If you have built-out comb, you could add those frames instead of adding back the extracted frames.
I know that beekeepers also add the frames back for a day or two even if they are reducing the hive just to clean up the frames, but this seems like a lot of work to go in the hive again.