I'm writing an X server in libcaca right now. It's great fun. https://github.com/robinpie/xcaca