Rosh Hashanah 4:4

Mishnah

At first, they would accept testimony regarding the new month the entire day. One time, the witnesses were delayed from coming, which caused the Levites to mess up in (their) song. They (then) decreed that they should not accept (the testimony) except until time of the Minchah (the time of the afternoon offering). (If they) came from the time of Minchah and onwards, they treat that day as holy (as Rosh Hashanah) and the following day as holy. Since the destruction of the Beit HaMikdash, Rabban Yochanan ben-Zakkai decreed that they (again) accept testimony on the new month the entire day.

Yerushalmi

[which caused the Levites to mess up in (their) song.]

Rebbi Shmuel bar-Nachman says: this is because of an event that occured when certain people (סרקייא) came and took the witnesses, causing the Levites to make a mistake in their song.

(Background on the song: The Levites would sing a specific chapter of Psalms to accompany communal offerings: the Tamid of the morning and the afternoon for each day of the week, as well as the additional offering on special days. This song varies by day of the week, and by holiday. Therefore, if a holiday falls on a Tuesday, for example, the Tuesday song would be replaced with the holiday song.)

Rebbi Acha bar Pappa said in front of Rebbi Zeira: The Chaveirim asked in front of Rebbi Shmuel bar-Nachman: In the name of whom did Rebbi (Yehudah HaNasi) say this (Mishnah)?

He said to them: When Rebbi (Yehudah HaNasi) gives over something without quoting a source, it is assumed to be Rebbi Yehoshua ben-Levi, and Rebbi didn't bring (in the Mishnah) the reason for this halachah (In other words, the essence of this "mess-up").

It is taught: On Thursday, the song is "Sing unto G-d, our Strength" (Tehillim 81:1), and on Rosh Hashanah it is "I removed the burden from his shoulder" (Tehillim 81:7) (This is from the same chapter, but starts later on in verse 7). If Yom Tov (of Rosh Hashanah) falls on Thursday, they would say in the morning "Sing unto G-d, our Strength)", and in the afternoon they would say "I removed the burden from his shoulder". If they would come after Minchah, they would say "Sing unto G-d, our Strength" (again), and they would end up repeating the chapter

(The repeating of the chapter is the "mess-up" referred to in the Mishnah)

[(If they) came from the time of Minchah and onwards, they treat that day as holy (as Rosh Hashanah) and the following day as holy.]

And why, if the witnesses haven't come by the time of Minchah did they keep the first day as holy? Because they have already assumed for the majority of the day that it is holy.

[Since the destruction of the Beit HaMikdash, Rabban Yochanan ben-Zakkai decreed that they (again) accept testimony on the new month the entire day.]

Rebbi Chiya bar-Ba said that Rebbi Yochanan would tell the people of a certain town that they should wait as long as possible (on the 30th of Elul) for the afternoon prayers, so that the prayers for Rosh Hashanah could be said on the the 30th of Elul and the 1st of Tishrei. That seems to be saying (see P'nei Moshe there) that if the witnesses don't come by the time of Minchah, we treat both days as holy, but wait for witnesses the entirety of the 30th of Elul in order to change the prayers appropriately.

Bavli (30b-31a)

In Bavel, it was explained that the "mess-up" was that no song was sung by the Levites at all. If we have a doubt as to if it is a holy day, we cannot sing the song for a regular day, and vice versa. Rebbi Zera (Rebbi Zeira from above) says that they sang the wrong song for the afternoon, as in the Yerushalmi. The conclusion in the Gemara is that there was no song at all.

The Bavli here says that the afternoon of Rosh Hashanah is chapter 29 of Tehillim, also different than the Yerushalmi there.