Please note, this answer was written in Februar 2015. The debate has significantly evolved since then, but I haven't gotten around to updating this answer, yet.

I'm sorry, this kinda turned out less brief than it started.

TL;DR: Very briefly, it is an issue of opposing ideologies:

Proponents wish to provide a common good to everyone and believe the increase in blocksize to be necessary to that end.

Opponents feel that a) it is impossible to introduce a change of that magnitude to Bitcoin at this stage, b) bigger blocks will mess up mining dynamics, or c) the increase changes properties of Bitcoin that shouldn't be changed.

What happened before

Originally, there was no blocksize limit. In fact, Satoshi Nakamoto was envisioning "100 million transactions per day, [...] taking 100GB of bandwidth" even in 2008. The blocksize limit was introduced later, when people where starting to play around with bitcoin, but before the value picked up significantly, when a worry came up that people might bloat the blockchain with cheap "spam transactions".¹

The problem is that 1MiB of transactions provides such a small capacity that, we'd be limited to seven transactions per second at minimal transaction size, or extrapolating from current transaction sizes, around four transactions per second. The block limit has not been encountered yet, and is therefore not yet a serious restriction, however this is significantly less than competing global payment networks can handle.

The thing is, that Bitcoin has become fairly valuable, and people want different things out of Bitcoin. The "apolitical money" is very political suddenly (as it always has been).

Arguments against increasing

There are quite a few different concerns here:

Bigger blocks will destroy the market for transaction fees

Increasing blocksize will make more room for transactions, which could reduce competition to be included in a block, in turn lowering transaction fees. In the long run, the block reward will dwindle, therefore less money would go towards mining, and the security of the Bitcoin network would wither with the lower income.

Bandwidth requirements are too much for full nodes

At 20MB per block, full nodes would have to download 2.8GB transaction data per day. This will be not only challenging data storage, but might actually be beyond the bandwidth capacity/datacap of some full node maintainers. One must consider especially that full nodes also serve the requests of thin-clients, so upload capacity might be more important than download capacity. However, on internet contracts for home-users upload speed is often significantly lower than download speed.

It has been suggested that a larger blocksize would quickly lead to a large increase of transaction numbers due to induced demand.

"Bigger blocks will lead to centralization."

Consensus may not be achievable

Once the first block larger than the 1MiB limit will be mined, a hardfork will occur between the network participants that refuse blocks greater than the limit, and network participants that accept it.

Blocks mined on the old network remain compatible to the new network, but not the other way around. Some voices have announced that they would use the fork as an opportunity to double-spend all their Bitcoins, spending them to themselves on the 1MiB-chain, and selling them on the 20MiB-chain, to profit and drive the new chain in the ground. Some users suggest that this supporters of that proposal have sufficient Bitcoin holdings and support that they can essentially force failure of the blocksize increase, they presume that exchanges would land on different sides in the discussion, but all users would quickly flock to the "safe old chain", once the "civil war" starts.

Bitcoin is destroying viability of altcoins

An increased Bitcoin blocksize would decrease demand for other blockchains, hurting investors of altcoins.

Bitcoin is not meant for every person on the planet to pay for their every cup of coffee

Some people feel that Bitcoin should be an exclusive privilege for settling between companies and the super-rich. A bigger blocksize would dilute the exclusiveness. They argue that something that is useful, but universally available is worthless. Bitcoin instead is meant to "force the poor to yield to the rich, universally, as a matter of course".

People are feeling that something is being decided without them being consulted

"There is only one proposal, we have no choice."

"We cannot predict how quickly bandwidth will grow, the proposed increase it too much."

"There is no consensus, forking without community support is a dead-end."

"There will be huge problems if we do this."

"Why haven't the miners been asked?"

I have trouble relating to these last statements, as the discussion about the blocksize limit has been going on for years.

Arguments for increasing

The transaction capacity is too low to support a global payment network

4.4 tps (transaction per second) are too few to support a global payment network.

At current network capacity, a bigger demand for transactions would cause regular users to be priced out of the blockchain. One would have to wait forever to have a transaction included in a block, and Bitcoin would eventually only get used to settle between banks, mayor corporations, and the super-rich. The blockchain instead should be accessible to everyone, and therefore the blocksize must be increased. Often, this is followed by the argumentum ab auctoritate, that the blocksize limit was always meant to be temporary, and the fork being necessary to achieve the vision Satoshi outlined in the original whitepaper.

Greater blocksize will increase total transaction fees

Even in a bigger block, transactions are not a free resource, as they cost bandwidth, data storage, and cause slower block propagation. Already, miners are not always including all transactions. Therefore, a market for transaction fees would exist even with bigger blocks, and more transactions would cause a higher total of transaction fees.

Effective blocksize will not increase over night

The blockchain of the past six years is smaller than 30GiB. It is wrong to assume that with the introduction of a larger limit, the blocks would suddenly fill up quickly, when they haven't done so before.

Consensus will be achieved before the hardfork is initiated

Major payment providers, exchanges, and mining pools will side with the hardfork supporters, causing the opponents to find themselves to be such a small minority that they will be stuck on an irrelevant alternative chain.

Technical issues will be fixed

Slower block propagation due to bigger blocks will be mitigated by Header-first synchronization and Inverted Bloom Lookup Tables. Internet connections will speed up sufficiently in parallel to increasing traffic demands on the network. Datastorage issues will be solved by introduction of a pruned blockchain, where most "full nodes" only keep a limited number of the last blocks, and few full nodes maintain the complete blockchain.

Conclusion

Personally, I think it is the natural progression of Bitcoin to increase the blocksize, as I see potential for Bitcoin to serve a broader audience. However, I do get some points of the contrarians, especially that it is hard to make predictions about how this will all play out.

Further reading is found here (and in a million other places):

About discussion:

Pro increase:

Contra increase:²

¹ Does anyone have links to the discussion on the introduction of the blocksize limit? I was looking for that.

² I'm still looking for more representative contra-positions. Wading through a few threads of some forums gave me a skin rash, but hardly anything useful.