Because of first sale doctrine. Anyone can set up a library by buying physical books, DVDs, or anything else they want to lend out to one person at a time without any sort of special permission. Digital library content, on the other hand, is based on specific contracts with the rights holders.
To be more precise, it doesn't *matter" if the Internet Archive is a "library" or not because libraries don't have any special status with respect to copyright law in the US. (Beyond whatever special status publishers may choose to give certain classes of libraries with respect to digital rights.) You or I can choose to setup a lending library tomorrow and we have the same rights to loan out physical books as the New York Public Library does.
