REST

I understand the main idea of REST is extremely simple. We have used web browsers for years and we have seen how easy, flexible, performing, etc web sites are. HTML sites use hyperlinks and forms as the primary means of user interaction. Their main goal is to allow us, clients, to know only those links that we can use in the current state. And REST simply says 'why not use the same principles to drive computer rather than human clients through our application?' Combine this with the power of the WWW infrastructure and you'll get a killer tool for building great distributed applications.

Another possible explanation is for mathematically thinking people. Each application is basically a state machine with business logic actions being state transitions. The idea of REST is to map each transition onto some request to a resource and provide clients with links representing transitions available in the current state. Thus it models the state machine via representations and links. This is why it's called REpresentational State Transfer.

It's quite surprising that all answers seem to focus either on message format, or on HTTP verbs usage. In fact, the message format doesn't matter at all, REST can use any one provided that the service developer documents it. HTTP verbs only make a service a CRUD service, but not yet RESTful. What really turns a service into a REST service are hyperlinks (aka hypermedia controls) embedded into server responses together with data, and their amount must be enough for any client to choose the next action from those links.

Unfortunately, it's rather difficult to find correct info on REST on the Web, except for the Roy Fielding's thesis. (He's the one who derived REST). I would recommend the 'REST in Practice' book as it gives a comprehensive step-by-step tutorial on how to evolve from SOAP to REST.

SOAP

This is one of the possible forms of RPC (remote procedure call) architecture style. In essence, it's just a technology that allows clients call methods of server via service boundaries (network, processes, etc) as if they were calling local methods. Of course, it actually differs from calling local methods in speed, reliability and so on, but the idea is that simple.

Compared

The details like transport protocols, message formats, xsd, wsdl, etc. don't matter when comparing any form of RPC to REST. The main difference is that an RPC service reinvents bicycle by designing it's own application protocol in the RPC API with the semantics that only it knows. Therefore, all clients have to understand this protocol prior to using the service, and no generic infrastructure like caches can be built because of proprietary semantics of all requests. Furthermore, RPC APIs do not suggest what actions are allowed in the current state, this has to be derived from additional documentation. REST on the other hand implies using uniform interfaces to allow various clients to have some understanding of API semantics, and hypermedia controls (links) to highlight available options in each state. Thus, it allows for caching responses to scale services and making correct API usage easily discoverable without additional documentation.

In a way, SOAP (as any other RPC) is an attempt to tunnel through a service boundary treating the connecting media as a black box capable of transmitting messages only. REST is a decision to acknowledge that the Web is a huge distributed information system, to accept the world as is and learn to master it instead of fighting against it.

SOAP seems to be great for internal network APIs, when you control both the server and the clients, and while the interactions are not too complex. It's more natural for developers to use it. However, for a public API that is used by many independent parties, is complex and big, REST should fit better. But this last comparison is very fuzzy.

Update

My experience has unexpectedly shown REST development to be more difficult than SOAP. At least for .NET. While there are great frameworks like ASP.NET Web API, there's no tooling that would automatically generate client-side proxy. Nothing like 'Add Web Service Reference' or 'Add WCF Service Reference'. One has to write all serialization and service querying code by hand. And man, that's lots of boilerplate code. I think REST development needs something similar to WSDL and tooling implementation for each development platform. In fact, there seems to be a good ground: WADL or WSDL 2.0, but neither of the standards seems to be well-supported.

Update (Jan 2016)

Turns out there is now a wide variety of tools for REST API definition. My personal preference is currently RAML.

How Web Services work

Well, this is a too broad question, because it depends on the architecture and technology used in the specific web service. But in general, a web service is simply some application in the Web that can accept requests from clients and return responses. It's exposed to the Web, thus it's a web service, and it's typically available 24/7, that's why it's a service. Of course, it solves some problem (otherwise why would someone ever use a web service) for its clients.