Short Answer

In terms of what they can carry, it's anything that's not a quest item. If you can drop it, store it, or barter it, you can give it to your follower.

In terms of how much they can carry, after some testing, I've determined that follower capacity is incredibly buggy: every time you switch followers, the weight capacity will go down.

So it starts out at 3001, and goes down from there. You can fix this by resetting your follower, either by the console (if on PC), or using the Wabbajack glitch (if on a console):

Long Answer

To test this, I tried testing with two followers: Lydia and Faendal.

Initial testing

I first took everything Lydia was carrying, which brought my carry weight up to 414. I then gave her stuff until the game refused to allow me to give her anything more, producing the error:

This item is too heavy to carry.

Once this occurred, everything on my side of the trade menu greyed out. My carry weight at this point was 180, which means Lydia maxed out at 234.

I then took everything Faendal was carrying, bringing my carry weight up to 491. Like I did with Lydia, I gave him stuff until I got the error message and my inventory grayed out.

He maxed out when my carry weight hit 207, which means his carry capacity was only 284.

Thinking this was kind of odd, I decided to try Lydia again, with my carry weight being 491 to start. Surprisingly, she maxed out when my carry weight hit 271, which means her carry weight was only 220 this time.

To ensure I covered all the bases, I cleared up my inventory so my carry weight was 414—as it was with Lydia the first time—and tested Faendal. He maxed out when I was at 144 carry weight, giving him a carry capacity of 270.

Testing a capacity scale factor

At this point, I had the idea that maybe the more stuff you have between you and your follower, the less they were willing to carry: that is, there's some negative scaling factor for follower capacity.

I decided to test Faendal's capacity, to see what that scaling factor was. To my surprise, I found it capped out at 269:

Total Weight | Faendal Capacity --------------------------------- 50 | 50 100 | 100 150 | 150 200 | 200 250 | 250 300 | 269 350 | 269 400 | 269 450 | 269 500 | 269

This contradicted my earlier findings.

Combined follower cap?

I'm then considered that there is capacity cap amongst all your followers, not just your current one.

If this is true, one of two hypotheses should be validated:

Lydia should be able to hold as much, if not more, than 269 weight. Once Lydia is stripped of everything, Faendal should be able to hold more than 269 weight.

I tested the first hypothesis: having stripped Faendal of all his possessions, I traded with Lydia several times, increasing the total weight each time. She capped out at 213:

Total Weight | Lydia Capacity ------------------------------ 50 | 50 100 | 100 150 | 150 200 | 200 250 | 213 300 | 213 350 | 213 400 | 213 450 | 213 500 | 213

So this hypothesis is incorrect.

That just leaves second hypothesis: if Lydia is stripped of everything, Faendal should be able to carry more than 269 weight.

Except that he didn't. He carried exactly 263 weight even after Lydia has been stripped.

What is this I don't even...

Random, or something else?

I thought maybe, just maybe, it's random. I decided to use the same set of items and same carry weight (289/300) and switch back and forth between Lydia and Faendal. This is what I found:

Round | Lydia Capacity | Faendal Capacity ------------------------------------------- 1 | 213 | 263 2 | 206 | 256 3 | 199 | 249 4 | 192 | 242

Well, that's odd: every time I switch, the follower capacity goes down by 7. Continuing on:

5 | 185 | 235 6 | 178 | 228 7 | 171 | 221 8 | 164 | 214 9 | 157 | 207 10 | 150 | 200

After 10 rounds, both Lydia and Faendal lost 63 weight capacity.

I got my programmer thinking cap on, and I wondered if it was a runtime issue: I decided to save the game, quit out, and reload.

No dice: Lydia and Faendal kept going down. Immediately after the save, it went down to 143 and 193 for Lydia and Faendal, respectively.

I then tried loading up an earlier save created before I did all these tests. Lydia shot back up to 234, and Feandal went back up to 284. These are the same as my original numbers.

One more round of follower trading, and both Lydia and Faendal went down to 227 and 277 capacity, as predicted.

Conclusion

Follower weight capacity is buggy as all get out. If you want to maximize your follower weight, do not switch between them.

Note 1: You can have your follower 'pick up' items without a weight capacity check, thus allowing your follower carry unlimited items.