I don't need to read this book (The Fair Tax) to know that I've just got two thoughts:
1. wow, it would be great to not have to worry about withholdings, filing taxes, and doing all that other annoying stuff.
2. Unfair burden on the poor. Percentage wise, the poor spend a much great portion of their income than do the rich (even if the rich buy extravagent stuff). This isn't always true, but it's a pretty fair generalization. There's only so much money one person can spend. Plus, there's a basic amount we all need to spend in order to survive, and that amount would be taxed evenly among all, even if someone was below the poverty level.
Now, if there were a way to exempt certain people from this large (23% is what the authors recommend, according to the blurb on amazon) sales tax (say, the un/underemployed and impoverished), then it might be worth considering. But I worry about the impact on the amount of money the country actually takes in. Even if it saves a lot by eliminating the IRS, you'd need to establish new departments to ensure the sales tax is being administered and collected properly (an IRS for just the businesses, if you will). Spending might also be negatively impacted, as some people might have a hard time spending almost 25% higher than the tag price on everything.