In addition to the Yankees who played there starting in 1923 (when Fenway Park had already been used for 9 seasons), there were many other uses. The New York "football" Giants played there from 1956 to 1973, Notre Dame football was played there a few times, three Popes gave speeches, it was the site of many famous boxing matches including the one when Joe Louis knocked out Max Schmeling in 1938, and there've been many concerts.
I've been to games there a few times - as a kid when I was mistakenly a Yankees fan and since then to see the Red Sox play them where I boldly wore my Red Sox cap and didn't get beat up or anything.
There are some people including a certain husband who think that this Yankee Stadium isn't really the original one (with the Babe Ruth design) because of the major reconstruction done in the 70's that among other things radically changed the dimensions. But both incarnations were the site of all these events. The announcers were trying to say the the new place will have the same Karma because of the fans, but it won't since it doesn't have the history.
The most remarkable speech ever was the one by Lou Gehrig who was retiring because of the onset of ALS, the disease that forever bears his name. "For the past two weeks you have been reading about the bad break I got. Yet today I consider myself the luckiest man on the face of the earth. I have been in ballparks for seventeen years and have never received anything but kindness and encouragement from you fans." Gehrig began. I've heard recordings of this many times. Gehrig who was a fantastic player was called the Iron Horse because he always played even when injured and held the record of 2,130 consecutive games played until it was bested by Cal Ripkin. He died two years after this speech.