The first complaint is code size, and it's not particularly important to me, compared to the second, but I'm only going to address two, and depressingly this is the one other people seem to care about more. Code size, Dean Edwards noted this problem with respect to Yahoo's library, a tree menu in 250kb (or maybe 100kb of compressed script) [*] Jeff Rafter recently wrote a treeview, it's 6k or so. I have no idea on the quality, but it's definately better than the yahoo one in that it degrades to HTML, and has drag'n'drop moving - it was just one I'd recently seen for the example.
QA, or the bigger problem
For me, the much bigger problem is QA'ing sites that are based on libraries, let's use a hypothetical example. You're creating a large site, 100 pages, you're using a library, it does 90% of what you want to do and is well supported, with frequent an irregular releases involving new features and bug fixes.
First up is what do you do about the 10%, to keep with the advantages of a library, you really need to integrate your new features into the library, which might be fine if the library developers accepted your changes into the core library, but we'll assume they don't (which isn't unreasonable, you'd probably not want the cost of making the code universal enough for a library anyway). This means you've now got a patched library to the core one available on the website.
Your code arrives at QA, hopefully an internal QA resource, but it could just be the public if you follow the no QA, who cares policy of web 2.0 companies. They find the inevitable bugs, you fix them, feeding the fixes back to the library developer and launch.
Of course if you've written the script yourself, you'll still have issues, but, the script will be smaller, it will do only what's necessary, the scripts will only touch the pages they're used in, so if you need to fix a flaw in your tree menu, the tree menu script will only impact those pages with tree-menus on, and there won't be any external dependency.
This concern mostly comes down to the fact that I always see QA as one of the hardest areas in a web project, QA is often underfunded, but even when not, recruiting good testers is tough, there's not that many about, and even ones who are good, often want to move into other areas - probably because of the underfunding. Libraries may reduce your development costs a little (although I'm not even convinced of this), but they certainly harm your testing cost.
A unit testing framework for the libraries could certainly change this, however the animation and CSS related issues that so often occure are very difficult to test, browser DOM's are not deterministic, so can't be stubbed out, equally often the DOM makes sense, but a bug elsewhere causes the rendering or interaction to fail, these sort of things mean you need humans involved, and humans cost money.
So use the libraries for prototyping, or quick and dirty short term projects, but stay away on the larger project unless your QA resoures are massive.