The point is that if you have a bad development process, people are going to have issues. And these issues are real.
Even though you have no "obligation" to solve these problems that you have created for other people, you shouldn't be surprised if people bring them up, or perhaps even fork the project with a bunch of other people who also have problems with you.
Sure, the community does not own your time, but neither do you own the community. A community is fully within it right to do something else with their time, or convince other people to contribute to a different project.