It's not just about being header-only, lots of libraries which aren't header-only still have code in headers. The library may choose to put certain functions in headers for performance reasons (to let compiler inline them), or, in C++, function templates and class templates generally have to be in headers.
But yeah, it's probably a good idea to not put code which breaks under -ffast-math in headers if possible.