Columns formed in 2003 as an instrumental act with Jason Skipper on drums, Jake Wade on bass, and Jake Troth on guitar. The band didn't have a singer because they couldn't find one. In 2004, Adam Cody from the band Glass Casket joined as vocalist. In 2006, they released a 3 song self-titled EP. Shortly after, Jake Troth left the band to go to school. After a short hiatus, John Stone joined and they recorded a 5 song EP in 2008. Stone left in 2009, and the band played a few shows with fill-in guitar player Sam Fleming (Zeus/Exploding Horse) for the next year.