As Alan observes, the "philosophy of facts" is really a philosophy of language: What do we really mean by "facts." Things get even more complicated when we consider how the concept "facts" and "scientific facts" only arise in Western history of science. There's no such thing as "facts" in Chinese history. There's also no immediately translatable term as well.
I've read Alasdair MacIntyre's After Virtue, and he discussed how the idea of "fact" arise during the modern period when scientific authority is at its climax. Note that this is also the period where German idealism is at its peak of development. The central problem is: how can be objectively quantify our experiences? Think of Descarte, Kant, and the like. How can we be sure that what I see as "red" is also what you see as "red?" - Something you've also discussed here.
This was indeed a philosophical crisis in the 19th and 20th century Europe. Before this intellectual turmoil, intellectuals had no issues believing that God will ensure that we're not delusional with our experiences. (Indeed, that was what Descarte proposed in Meditations.) So, to fill up this epistemic gap, people resorted to a seemingly theological and authoritative concept of "facts."
This isn't surprising. We tend to use "facts" like its the word of God. "Are you going to argue with facts?" "Look at the facts! You can't possibly deny the truth!"
Many contemporary postmodern philosophers realize this. That's why you have philosophers like Nietzsche and Wittgenstein arguing that "facts" cannot ground objectivity. Because historically, they realized that "facts" is a tape that covers the gap in our epistemic grounding. It didn't patch it up properly.
In the end, I'm at the same aporic state as you are. But the history of "facts" is interesting as it shows us its a product of an intellectual struggle for epistemic grounding.