Yes, June 2 is a national holiday, schools and most public offices will be closed even though it is a Thursday. Most places reopen the day after, but those that can, will take advantage to take a 4 day long weekend ;-) That just means that you will likely find a busy weekend with lots of other travelers taking advantage of the long weekend.
Most museums never close for June 2, the "festivities" are all official ceremonies in public buildings with ceremonies. Sometimes there might be a special event - a few years ago we have the Frecce Tricolori do a fly over Florence for the day (they are the specialized team of 9 jets that do shows in formation). Since you'll be in Rome for that day, it is more likely you will find some type of special parade or ceremony somewhere around the Senate or Presidential palaces, but I am not that well informed on what goes on in Rome for the holiday other than to say that since it is the capital, they for sure will have something going on. If you are interested in visiting specific museums in Rome, look for their official websites and check whether they have "closed" dates. Most likely June 2 will not be on there, it means it will be open.
If you're trying to decide whether to be in Florence or Rome on that day, to see something interesting, it is more likely you will be satisfied with what they do in Rome.