Let's start by distinguishing between vanity publishing and self-publishing. I'm sure some of you learned the difference as a result of a certain recent kerfuffle, but in case you didn't:
Vanity publishing: Sometimes also called subsidy publishing, though there are technical differences between the two. More or less, you pay for all aspects of the publication of the work. The vanity company offers a variety of options, which you select as from a menu. You get no services other than those you pay for. You get little or no meaningful distribution through the label, with little to no possibility of distributing it yourself through standard channels. The ISBN and bar codes are linked to the vanity label, not to you. You give up part or all of the rights, and you don't get the profits but are paid some form of royalty or split.
Self-publishing: You act as your own publisher. You hire a press, a designer, editors, artists, and so on, or you do the work yourself. You learn about things like bleeds and trims, and you spend many hours wondering just how necessary Adobe Distiller really is. You get an ISBN of your own, and maybe bar codes, too. (Though, if you're frugal, you might get these in other ways than by registering your company and paying the fees.) You keep all rights to the work (through your company, probably), and you keep all profits from sales. You figure out distribution options and, if you're lucky, sell to national accounts.
There are legitimate reasons for choosing either option. If you have little publishing experience and less time, and if you have a project of mostly personal value -- a family history, your dissertation, your child's drawings and notes -- vanity publishing might be a good option. If you have a nonfiction project and an established platform, and you're willing to learn and work, then self-publishing might be a good option.
Generally, neither is a good option for fiction. There are exceptions to that general principle, of course, and it may be that you had some legitimate reason for choosing one of these options, some of which I'll mention below.
Which brings us to your query letter.
If you have self-published a non-fiction book, it won't hurt you at all to mention that in your query. It might not be necessary as it might not fit your query, but that is rather case-specific. The point is, if you choose to include it, it probably won't hurt you. You can mention something about how you've built a platform for your nonfiction works, have experience with PR and marketing, that sort of thing. You might even mention how you could leverage your existing non-fiction platform and press contacts for your novel's publicity.
If you have self-published a novel or other fiction, you will probably want to keep that fact to yourself unless you can legitimately demonstrate a solid reason for self-publishing. Without such a reason, I will assume you self-published because you couldn't find someone else to publish it for you. And that won't score you any points.
Examples of legitimate reasons:
- You wanted to learn the publishing business from every angle.
- You ran your own press and published many authors, including yourself.
- You were mid-series when your publisher folded and you wanted to complete the series yourself.
- You have an extensive publication history and recognized that this was a pet, one-off project not worth troubling your publisher with.
What do these examples have in common? They don't undermine your credibility as a professional. In fact, they might even enhance it by demonstrating your understanding of the industry.
If you have vanity published a personal project, it probably won't hurt to mention it as long as it's clear that this was a highly personal project suitable for vanity publication. But is it relevant? Why would your decision to present your siblings with bound copies of grandpa's journal be relevant to our potential business relationship? I'm not saying it can never matter -- perhaps this was the project that whet your appetite for publishing and led you to try your hand at writing. But if there's no connection, there's probably no reason to mention it.
If you have vanity published a novel, you're probably better off keeping that information to yourself. Generally, we expect novels to be published through normal market channels *if* they're good enough. If they're not good enough, and they get rejected by everyone on the planet, that's not exactly a fact you would want to broadcast.
Now it may be that you have some legitimate reason for vanity publishing a project that isn't quite a personal project. I'm willing to believe it's possible, even though I'm not personally aware of any such reasons. In that case, tell me the reason in the query letter. Otherwise, I will assume the book just wasn't good enough and you vanity published as an act of desperation. Sorry if that sounds cold. Sometimes the truth, she is an icy bitch.
What's the bottom line? Have a good reason to justify it, or don't mention it at all.