While the history of taxonomic diversification in open ocean lineages of ray-finned fish and elasmobranchs is increasingly known, the evolution of their roles within the open ocean ecosystem remains poorly understood. To assess the relative importance of these groups through time, we measured the accumulation rate of microfossil fish teeth and elasmobranch dermal denticles (ichthyoliths) in deep sea sediment cores from the North and South Pacific gyres over the past 85 million years. We find three distinct and stable open ocean ecosystem structures, each defined by the relative and absolute abundance of elasmobranch and ray-finned fish remains. The Cretaceous Ocean (pre-66 Ma), was characterized by abundant elasmobranch denticles, but low abundances of fish teeth. The Paleogene Ocean (66-20 Ma), initiated by the Cretaceous/Paleogene Mass Extinction, had nearly 4 times the abundance of fish teeth compared to elasmobranch denticles. This Paleogene Ocean structure remained stable during the Eocene greenhouse (50 Ma) and the Eocene-Oligocene glaciation (34 Ma), despite large changes in overall accumulation of both groups during those intervals, suggesting that climate change is not a primary driver of ecosystem structure. Dermal denticles virtually disappeared from open ocean ichthyolith assemblages about 20 Ma, while fish tooth accumulation increased dramatically in variability, marking the beginning of the Modern Ocean. Together, these results suggest that open ocean fish community structure is stable on long timescales, independent of total production and climate change. The timing of the abrupt transitions between these states suggests that the transitions may be due to interactions with other, non-preserved pelagic consumer groups.