Depending on the design and chemistry of your lithium cell, you may see them sold under different nominal "voltages". For example, almost all lithium polymer batteries are 3.7V or 4.2V batteries. What this means is that the maximum voltage of the cell is 4.2v and that the "nominal" (average) voltage is 3.7V . As the battery is used, the voltage will drop lower and lower until the minimum which is around 3.0V. You should see the number 3.7V written on the battery itself somewhere.

For example, here is a profile of the voltage for a 'classic' 3.7V/4.2V battery. The voltage starts at 4.2 maximum and quickly drops down to about 3.7V for the majority of the battery life. Once you hit 3.4V the battery is dead and at 3.0V the cutoff circuitry disconnects the battery (more on that later.

You may also run across 4.1V/3.6V batteries. These are older than 4.2V/3.7V - they use a slightly different chemistry and you'll see the 3.6V marking on the cell.

Nowadays you may also be able to purchase 4.35V cells! These are the latest chemistry, they have a little more power as indicated by the voltage being higher than 4.2V. They tend to be cylinder lithium ion's used for laptop batteries, and lights so its not terribly likely you'll just run into one unless you're looking for it. Make sure when you're buying batteries and chargers to match them up! Overcharging a 3.6V battery by attaching it to a 4.2V battery charger can at the very least permanently damage your battery and at worst cause a fire!