QA and Javascript Libraries
Javascript libraries have become increasingly common, there's a whole plethora to choose from, with a different range of feature sets. They allow you to rapidly prototype something, but I will argue below that they shouldn't yet be used in a real environment. If you've followed my usenet discussions, or got me on the topic over a beer or on IRC, you'll know everything below, the themes have been knocking around my head for a long time, Andrea Campi finally triggered the post.
The first complaint is code size, and it's not particularly important to me, compared to the second, but I'm only going to address two, and depressingly this is the one other people seem to care about more. Code size, Dean Edwards noted this problem with respect to Yahoo's library, a tree menu in 250kb (or maybe 100kb of compressed script) [*] Jeff Rafter recently wrote a treeview, it's 6k or so. I have no idea on the quality, but it's definately better than the yahoo one in that it degrades to HTML, and has drag'n'drop moving - it was just one I'd recently seen for the example.
For most projects, the javascript is basically incidental, it's a little effect, or a little piece of user interface like a tree view, that improves the page a little, these are simple, they can be written quickly - re-using code from "a copy and paste library" - and do just what you want and no more, which means they are a fraction of the size for the user to download. With so many of these libraries waiting until onload before they can do anything, this is particularly relevant - the user spends there whole time waiting to interact with the page until everything has downloaded, it may work great when using it locally or on your business adsl, but it's a nightmare on a broadband connection that's saturated with p2p data.
QA, or the bigger problem
For me, the much bigger problem is QA'ing sites that are based on libraries, let's use a hypothetical example. You're creating a large site, 100 pages, you're using a library, it does 90% of what you want to do and is well supported, with frequent an irregular releases involving new features and bug fixes.
First up is what do you do about the 10%, to keep with the advantages of a library, you really need to integrate your new features into the library, which might be fine if the library developers accepted your changes into the core library, but we'll assume they don't (which isn't unreasonable, you'd probably not want the cost of making the code universal enough for a library anyway). This means you've now got a patched library to the core one available on the website.
Your code arrives at QA, hopefully an internal QA resource, but it could just be the public if you follow the no QA, who cares policy of web 2.0 companies. They find the inevitable bugs, you fix them, feeding the fixes back to the library developer and launch.
Three months later, and want to add a couple of new features to one of your pages, the library has had some changes, this is where the problems arrive. The 10% it doesn't do is now 5%, but they've solved it differently to you did, the bugs you fixed are integrated, and so are a number of others. What do you do? If you change the library to the new one, every one of your 100 pages needs to go through QA again, the impact of changing the library code has an immense cost in time to retest everything, this is of course especially relevant if your fixing a security flaw. Yahoo have already shown they can't write javascript without security flaws.
Of course if you've written the script yourself, you'll still have issues, but, the script will be smaller, it will do only what's necessary, the scripts will only touch the pages they're used in, so if you need to fix a flaw in your tree menu, the tree menu script will only impact those pages with tree-menus on, and there won't be any external dependency.
This concern mostly comes down to the fact that I always see QA as one of the hardest areas in a web project, QA is often underfunded, but even when not, recruiting good testers is tough, there's not that many about, and even ones who are good, often want to move into other areas - probably because of the underfunding. Libraries may reduce your development costs a little (although I'm not even convinced of this), but they certainly harm your testing cost.
A unit testing framework for the libraries could certainly change this, however the animation and CSS related issues that so often occure are very difficult to test, browser DOM's are not deterministic, so can't be stubbed out, equally often the DOM makes sense, but a bug elsewhere causes the rendering or interaction to fail, these sort of things mean you need humans involved, and humans cost money.
So use the libraries for prototyping, or quick and dirty short term projects, but stay away on the larger project unless your QA resoures are massive.
The size is made worse of course by the fact yahoo's developer resources aren't cacheable, particularly bad for a company that employees the author of the cacheability engine.
Thu May 25 15:57:52 UTC+0100 2006 I hope what you're saying is that the JavaScript frameworks (and possibly browser JS engine capabilities in terms of library/module loading) are not mature enough yet, because I still firmly believe that we should eventually be moving up the development stack in the web development community. Libraries and frameworks are a sign that a language is starting to mature enough that people want to do even more complex things in the language. Have you considered intellectual property aspects? If I custom-develop some JavaScript widgets for Company A, I can't reuse what I've learned for Company B because Company A owns that code. Standardizing on libraries or frameworks helps can help in that respect. Anyway, if I had to write/debug/test my own hashmap or stack in C++/Java every time I started a new project, productivity would dip and I'd go insane. It's great that I can rely on the STL/Boost or the Java SDK for this.
Thu May 25 17:21:02 UTC+0100 2006 yahoo's developer resources are cacheable. the production releases all reside under the yimg.com domain. yahoo is guilty of a lot, but cache-friendliness is not one of them. compare their expires headers, division of delivery to honor http limitations, use of akamai, etc. this is not excusing bloated or buggy code though.
Thu May 25 17:38:31 UTC+0100 2006 grumpY! Check the link to the cacheability engine, the Yahoo developer resources there are not cacheable.
Thu May 25 18:58:42 UTC+0100 2006 jim, lets pull a random yahoo JS lib and put this issue to rest: http://us.js2.yimg.com/us.js.yimg.com/lib/common/utils/2/connection_2.0.0-b4.js a beta version of one of their files this is being served via a domain that has akamai coverage this has expiration headers in 2016 the url contains no gibberish or unique metadata that might impair caching if your tool says this cannot or is not cached, your tool is wrong
Thu May 25 20:06:48 UTC+0100 2006 grumpY! - the URL I get from the tree menu page is http://com1.devnet.scd.yahoo.com/yui/build/connection/connection.js not, one of the yimg ones, and it's not cacheable, it's not my tool, it's the excellent cacheability engine from Mark Nottingham which says it's not cacheable, but any simple inspection shows it's not cacheable. I dunno what pages you're looking at, but I'm looking at http://com2.devnet.scd.yahoo.com/yui/examples/treeview/default.html?mode=dist which is the url the search engines provide for it, if alternatively I use http://developer.yahoo.com/yui/treeview/ which appears to be the same http://www.web-caching.com/cgi-web-caching/cacheability.py?query=http%3A%2F%2Fdeveloper.yahoo.com%2Fyui%2Ftreeview%2F&descend=on then I get similar results. So whilst I agree the yahoo site itself is very good at organising caching, the developer site doesn't appear to be at all, where did you get the link above from?
Thu May 25 21:55:35 UTC+0100 2006 Yeah, the developer site is not a fair example of the use in the wild, it's bloated because of comments too, which you'd strip as part of a release to testing hopefully. I didn't really mean to pick on yahoo's library, it's certainly very good, it was just easiest as Dean Edwards had already done the effort on the size of the library. So don't think I'm criticising the yahoo library in particular, it applies to all libraries, the yahoo one is one of the better libraries itself, my advise is do no more than demos and prototypes with it though.
Thu May 25 21:36:40 UTC+0100 2006 first of all i did not mean to insult the author of the caching tool. if that was perceived i apologize. the link was one i pulled internally from the development site for these libs. i don't mind posting it here because i don't believe that it is considered "secret", these new versions are set to be delivered shortly, and the yimg urls are used internally (as they are used on any non .html files on yahoo's sites, or should be) anyhow, i now see where you are getting your file, from the "official" ydn site. for example http://developer.yahoo.com/yui/build/yahoo/yahoo.js which does NOT have correct caching, as you point out. you are right and i am wrong. the only consolation i can provide is that the ydn site appears to strongly suggest that users serve these files from their own site....where maybe they can implement proper caching.
Thu May 25 22:11:16 UTC+0100 2006 Jeff, Libraries and frameworks are a sign that a language is starting to mature enough that people want to do even more complex things in the language. That depends, it doesn't actually mean you should be doing more complex things in the language, it may be that the language isn't right for doing these more complicated things - like writing an operating system in VB, it's not going to help no matter how many libraries you bring to the party. Licensing of code snippets isn't a problem I think, if your snippets are large enough to carry any IPR, then you're not following my maxim of minimum code required to the specific task you have, or you are doing very, very similar tasks each time - if that's the case then certainly try to get the code opened up so you can licence it to each client, but I don't see the premise that starting with a library and building your 10% actually makes any difference, you'll still be building your same 10% for the next customer, and the customer will have all the same size and QA issues. Certainly changing javascript into some object orientated language with packages would help in some ways, but it doesn't solve every issue.