Write the book you want?

The most common advice I hear as a writer is to "write what you want to write."
Don't write to a formula.
Don't write for the market.
Don't write what you think agents and publishers might want.
Just write the book you want to write.

This is exactly what I've been doing and I'm on my third book now. I wrote what I wanted and didn't consider whether they were commercial or whether there was a market or any of that stuff. I wrote them because I was swept away by an idea and I wanted to find out what happened next.
But...
What if what you want to write is not commercial or marketable or popular?
What if you don't have a killer concept or a breakout idea that's never been done?
What if the book you want to write is not the book an agent or publisher will want to take on?
What if you spend a year or more on an idea that will never sell?

Are we setting ourselves up for failure by assuming we can write what we want and be able to sell it? Should we be thinking about our work as more than just a way to express ourselves?

There seem to be more questions than answers in this blog post and that's because I'm afraid I don't really know the answer!

I think that what the advice really means is that only when we allow ourselves creative freedom will we ever write anything worthy of being published. Formulaic writing to try and cash in on the latest vampire craze or  dystopian mania will never ring as true as something we write from our heart.
I think that's true but I worry nonetheless.

What do you think?