The answer might seem a little counter intuitive at first but we'll see...
The short answer is that it is in your advantage to exchange. But why?
Well initially there was a 1/3 chance that you were holding the envelope with the note in it and a 2/3 chance that the note was on the table.
This is still the case after one of the envelopes on the table has been removed, there is still a 1/3 chance that you have the note and a 2/3 chance of it being on the table.
If this is confusing then it may help to think that the questioner knows which envelope the $20 note is in, though in practice it doesn't actually matter. The questioner would always be able to demonstrate that the note was not in one of the envelopes on the table regardless of where the note was, so the fact that he was able to do this changes nothing.
Consider a different example....
Say there are a 1000 envelopes on the table, 1 with a note inside. You pick 1 envelope, the chance that this has the note in it is clearly 1/1000, where as the chance that it is still on the table is 999/1000. Odds are its on the table. Now the questioner could, assuming he can remember where the note is demonstrate to you that the note is not in 998 of the envelopes on the table. In this case nothing would have happened to change the fact that there is only a 1/1000 chance of you having the note.
That is why you exchange.
What is the value of the exchange?
Simply before the exchange you have 1/3 of $20 and afterwards you will have 2/3 of $20, ie the advantage to you is about $6.66