A "challenge" is the process by which a lawyer asks that a juror be excused. There are two types of challenges:
Challenges for a Cause - Here the lawyer claims that the juror might not be able to render an impartial verdict. For example, a lawyer asks to excuse a juror from an auto theft trial because the juror had a car stolen before. The judge may grant or deny a challenge. There is no limit to the number of challenges for a cause that a party can make, but there must be a reason for the challenge.
Peremptory Challenges - Each side has a limited number of challenges for which no reason need be given. These peremptory challenges give both sides some choice in the make-up of the jury.
The fact that you are not chosen to be on the jury is not a reflect on your integrity or worthiness to serve.