There are large and complex packages that were written this way: use a notebook (including text cells and comments) and auto-save to an .m file. SciDraw is one example. One nice feature of notebook is hierarchical sectioning.
The reason why I don't like to do this personally (for packages!) is that it's hard to use a version control system with it. The missing piece is a good reliable notebook diff tool, I guess. There is in fact a diff tool in Mathematica, but in the end I found using plain text more convenient for packages.
I use the Mathematica plugin IntelliJ IDEA instead of Workbench for packages.
But interactive work is something entirely different. IDEs are just not suitable for it. Even when writing packages, I constantly have a notebook open to interactively test it, even put new functions together within the notebook before moving the code to the package file.