The Croatian government had alleged that Serbia committed genocide in the town of Vukovar and elsewhere in 1991.
Serbia later filed a counter-claim over the expulsion of more than 200,000 Serbs from Croatia.
About 20,000 people died during the 1991-1995 war, mostly Croatians.
The Croatian town of Vukovar was devastated when it was occupied by Serbs for three months in 1991. Tens of thousands of ethnic Croats were displaced, and about 260 Croat men were detained and killed.
Four years later, the Croatian military’s Operation Storm bombarded the majority ethnic-Serb Krajina area, forcing about 200,000 people from their homes.
Speaking in court on Tuesday, Judge Peter Tomka dismissed both the Croatian claim and the Serbian counter-claim.
Forces on both sides had carried out violent acts during the war, Judge Tomka said. However, neither side had provided sufficient evidence to demonstrate the "specific intent required for acts of genocide".
BBC News Reported:
The court’s ruling was posted online.