Misconception:
Swimming on Shabbos or Yom Tov is prohibited because one might come to build a raft.
Background
There is a Mishnah that states that swimming on Shabbos or Yom Tov is prohibited, and the Gemara explains that this is a preventative measure lest one come to build a raft (Beitzah 36b)1. The Gemara elsewhere, however, relates the following story:
א"ר זירא אנא חזיתיה לר' אבהו דשט באמבטי ולא ידענא אי עקר אי לא עקר פשיטא דלא עקר דתניא לא ישוט אדם בבריכה מלאה מים ואפי' עומדת בחצר לא קשיא הא דלית ליה גידודי הא דאית ליה גידודי
R. Zeirah said: I saw R. Abbahu swimming in a body of water, but I do not know whether he “lifted” or not. Is it not obvious that he did not “lift”? For it was taught: One must not swim in a pool full of water, even if it stands in a courtyard. There is no difficulty: in the one case it [the pool] has no borders; in the other case it has.
Ritva explains that “lifting” refers to whether or not R. Abbahu had splashed any water out of the pool as he swam. This Gemara explains that the Mishnah’s prohibition of swimming on Shabbos or Yom Tov applies only when the body of water does not have any clear borders. Considering that the pool that R. Abbahu swam in had borders, there was no issue whether he splashed or not, nor was there a concern about any raft building.
Thus, according to this, backyard pools of today seem to be perfectly permissible as they have borders. The prohibition in the Mishnah of swimming on Shabbos or Yom Tov lest one come to build a raft would apply only to borderless bodies of water — like lakes, rivers, oceans — in which it is not at all strange to presume one might come to build some sort of flotation device. Indeed, the Shulchan Aruch (O.C. 339:2) rules that there is no concern about building a raft when swimming in a bordered body of water on Shabbos or Yom Tov.
Modern Pools
So why, then, is it prohibited nowadays to go swimming in a typical bordered pool on Shabbos or Yom Tov?
There are a number of issues. For one, the Gemara (Shabbos 113b) prohibits fully soaking any clothing in water on Shabbos or Yom Tov2, a ruling codified by the Shulchan Aruch and Rema (O.C. 302:9). R. Moshe Feinstein (E.H. Vol. II, 13) also points out that there is a very real, and rather common concern that one might come to wring out his bathing suit, which would be a violation of the Biblical prohibition of sechitah. Out of fear of this, swimming on Shabbos or Yom Tov would be prohibited.
Beyond these two rather formidable concerns, the largest issue with swimming on Shabbos or Yom Tov is quite straightforward: There is a Rabbinic prohibition of bathing, or submerging oneself, in heated water on Shabbos or Yom Tov. This prohibition is codified in the Shulchan Aruch (O.C. 326:1), and the Mishnah Berurah (9, 21) extends this ruling to even cold water as has been the accepted stringency of the Trumas HaDeshen for centuries. R. Moshe Feinstein (O.C. Vol. IV, 75) also explicitly rules that entering into a pool of water of any temperature would be categorically forbidden. Indeed, even if one were to (unadvisedly) ignore the rulings of the Mishnah Berurah and R. Moshe Feinstein, swimming would still only be permitted in the nude, and in cold water.
Certainly, if the water is heated even the least bit, swimming on Shabbos or Yom Tov would be absolutely prohibited according to all authorities — regardless of one’s raft-building capabilities.
For further discussion on this topic, and the place from which I compiled the above sources, see this shiur given by R. Aryeh Lebowitz.