µstr - Micro String API - for C

Download - Version µstr-1.0.4

You can get the stable release of 1.0.x here (ftp), it still doesn't have 100% documentation but it's pretty close. You can get the current git tree here.

About

A few years ago now I wrote a very extensive String API for C, called Vstr, it was designed to perform extremely well for IO like patterns as that was my planned usage (for instance And-httpd, my Web server). It works very well, for that usage.

Also due to the extensivness of the API I basically used it everywhere, even though there are some things it is somewhat "overkill" for, and I wanted other people to use it so I didn't have to resort to using string.h when creating patches for their code. However more than a few C coders I speak to have one of a few reasons why they don't want to use Vstr. The ustr API should solve all of these problems, and hopefully fill in all the gaps where Vstr is the 500lb hammer.

There is an introduction to using the ustr API, as a guided tutorial of example code. You can see the developers design documentation here. Also here's the file of GDB init functions (or: ustr-import gdb), to help with debugging. There are API references for functions and constants. The unit testing coverage data is available. And the here are links to TODO, NEWS, THANKS, AUTHORS and the "LICENSE", such as it is.

List of problems with String APIs, in C

These are the primary reasons I've seen given why developers don't want to put "any" string API into their code:

It's "too hard", politically, to put into an existing projects build system.

String ADTs take up too much memory.

It's too hard to integrate into existing code/APIs that uses plain C-style strings significantly.

We can "often" use stack space for a string, if we do it ourself, which (we understand) is much faster.

Need/want to initialize strings as they are declared.

License isn't compatible with our code.

The current code may be unsafe and hard to audit, but it's "optimized" to use the C-style string format/design.

While development might be harder "debugging is easier".

So instead of using a String ADT (any String ADT), the common solution is to just try and get by using (char *)'s and string.h. With "certain bits" using strdup() and/or some size_t's with a stored length. This ad hoc approach is very bad, for everybody. Needless to say, ustr solves all of the above problems while being significantly faster and safer than just using string.h.

List of solutions to problems with String APIs, in C

Solving the build problem

To use ustr with all of the shipped modules, you can just (after install ustr) goto your source dir. and run:

ustr-import all

And include the following line in any source files you want to use ustr from:

You do not have to worry about linking with anything or having symbols appear in your library that you don't want ... it will "just work". If you don't want copies of each function you are using to appear in each file it is used from you can pass the "-c" option to ustr-import and compile the .c files. You can also link against the ustr shared library, if you don't mind adding a very simple dependency (it is much simpler than Vstr, for example, and so is very likely to just run anywhere your code does).

Note that if you wish to look at some ustr using code now, you can see some example code or look at the distributed example/unit-test code at the end of this page.

Solving the memory overhead problem

This is best illistrated by example. Say you build a "simple" string API, it just has the basics: automatic grow and shrink and reference counts for zero copy duplication. The default assumption is that as you are adding data to the string you don't want to spend all your time in realloc() so you also expand the size of the string by powers of 2.

This is about the most common string API that people implement on their own, a rough structure would look like:

struct Simple_str { char *ptr; size_t reference_count; size_t length; size_t size; };

However this strucutre itself is 16 bytes (in a 32bit environment) or 32bytes (in a 64bit environment), and to allocate a new string requires two allocations (one for the string, and one for the data in the ->ptr argument).

This means that to create a new string containing one byte you have to do two allocations and allocate roughly 34bytes ... giving a 1700% increase over plain strdup().

By comparison ustr does one allocation of 6bytes by default, in a both 32bit and 64bit environments, and it can drop that down to 4 bytes, again even in a 64bit environment (and, in a 64bit environment, you can have a > 4GB ustr so you aren't losing any features).

Although, to be fair, by default ustr only allocates 2bytes for it's reference count. However that's 65,535 references to the same string -- and ustr has a helper function so that if you need 70,000 references to a string it'll only need to allocate 2 strings, and has another helper function so that you can mark a string as "shared" so it will act like a read-only string until it has been set back to "owned".

For a Gnumeric spreadsheet comparing all string length values upto 260 between strdup(), ustr and a "Simple str" click here. I've also listed of some common "small" string lengths below with their percentage overhead vs. strdup().

Ave. range of string sizes Ave. Simple_str - 32bit overhead Ave. Simple_str - 64bit overhead Ave. ustr overhead 1-9 bytes 327.27% 618.18% 85.45% 1-19 bytes 194.76% 347.14% 55.24% 1-29 bytes 239.57% 236.34% 45.81% 1-39 bytes 123.78% 201.83% 41.22% 1-79 bytes 87.01% 126.51% 32.53% 1-89 bytes 83.13% 118.29% 28.3% 1-198 bytes 62.23% 78.23% 23.85%

Here is a list of the first 18 string lengths, and some later ones using the different options for ustr:

String Alloc. C-string Alloc./Used Ustr, ref==4 Alloc./Used Ustr, ref==2 Alloc./Used Ustr, ref==1 Alloc./Used Ustr, ref==0 "" 1 byte 0/0 0/0 0/0 0/0 "1" 2 bytes 8/8 6/6 6/5 4/4 "12" 3 bytes 12/9 8/7 6/6 6/5 "123" 4 bytes 12/10 8/8 8/7 6/6 "1234" 5 bytes 12/11 12/9 8/8 8/7 "12345" 6 bytes 12/12 12/10 12/9 8/8 "123456" 7 bytes 16/13 12/11 12/10 12/9 "1234567" 8 bytes 16/14 12/12 12/11 12/10 "12345678" 9 bytes 16/15 16/13 12/12 12/11 "123456789" 10 bytes 16/16 16/14 16/13 12/12 "123456789 " 11 bytes 24/17 16/15 16/14 16/13 "123456789 1" 12 bytes 24/18 16/16 16/15 16/14 "123456789 12" 13 bytes 24/19 24/17 16/16 16/15 "123456789 123" 14 bytes 24/20 24/18 24/17 16/16 "123456789 1234" 15 bytes 24/21 24/19 24/18 24/17 "123456789 12345" 16 bytes 24/22 24/20 24/19 24/18 "123456789 123456" 17 bytes 24/23 24/21 24/20 24/19 "123456789 1234567" 18 bytes 24/24 24/22 24/21 24/20 "123456789 12345678" 19 bytes 32/25 24/23 24/22 24/21 "123456789 123456789" 20 bytes 32/26 24/24 24/23 24/22 "123456789 1234...45" 26 bytes 32/32 32/30 32/29 32/28 "123456789 1234...56" 27 bytes 48/33 32/31 32/32 32/31 "123456789 1234...78" 28 bytes 48/34 32/32 32/31 32/30 "123456789 1234...89" 29 bytes 48/35 48/33 32/32 32/31 "123456789 1234...9 " 30 bytes 48/36 48/34 48/33 32/32 "123456789 1234... 1" 42 bytes 48/48 48/46 48/45 48/44 "123456789 1234...12" 43 bytes 64/49 48/47 48/46 48/45 "123456789 1234...23" 44 bytes 64/50 48/48 48/47 48/46 "123456789 1234...34" 45 bytes 64/51 64/49 48/48 48/47 "123456789 1234...45" 46 bytes 64/52 64/50 64/49 48/48 "123456789 1234...89" 60 bytes 96/66 64/64 64/63 64/62 "123456789 1234...9 " 61 bytes 96/67 96/65 64/64 64/63 "123456789 1234... 1" 62 bytes 96/68 96/66 96/65 64/64 "123456789 1234...23" 124 bytes 192/130 128/128 128/127 128/126 "123456789 1234...34" 125 bytes 192/131 192/129 128/128 128/127 "123456789 1234...45" 126 bytes 192/132 192/130 192/129 128/128 "123456789 1234... 1" 232 bytes 256/256 256/254 256/253 256/252 "123456789 1234...12" 233 bytes 256/256 256/254 256/253 256/252 "123456789 1234...89" 250 bytes 256/256 256/254 256/253 256/252 "123456789 1234...9 " 251 bytes 384/258 256/255 256/254 256/253 "123456789 1234...45" 256 bytes 384/263 384/261 384/260 384/259 "123456789 1234...56" 257 bytes 384/265 384/263 384/262 384/261

Solving the integration problem

This problem is mostly solved by having ustr's store their data as a valid C-style string at all times (although it can have NIL bytes in it, which will confuse other APIs if you only pass the ustr_cstr() value.

Also as you will see below it is trivial to "port" designs using common methods to use ustr.

Solving the stack allocation problem

There is a simple solution to this problem, ustr strings can be allocated on the stack. For example:

char buf_store[USTR_SIZE_FIXED(1024)]; Ustr *s1 = USTR_SC_INIT_AUTO(buf_store, FALSE , 0 );

The second argument says if the ustr should should be of limited size (and so fail if too much data is added to it) or not and thus it should automatically be converted to heap based storage if you try and add too much data to it (and thus needing to be ustr_free()d). So you can still do that, if you think you need to ... and it'll now be managed, faster and safer.

You can call ustr_free() on ustr's on the stack, and they will not be passed to free. Also if they get duplicated the duplication comes from the heap (you cannot create references to a fixed ustr).

Solving the initialization problem

There is a simple solution to this problem, ustr strings can be constant C-strings. For example:

Ustr *s1 = USTR( "" ); Ustr *s2 = USTR1(\xF, "123456789 12345" ); Ustr *s3 = USTR1(\x18, "123456789 123456789 1234" ); Ustr *s4 = USTR2(\x1, \x44, "1234589 12" ... "89 1234" );

As with the stack based ustr's, calling ustr_free() doesn't actually free these strings. Also as they are read-only strings, calling ustr_dup() on them will just return the pointer itself (Ie. no allocations happen). Apart from that they act as ustr's in every way. This means that you can initialize things to default values in other structs/arrays, and then add/delete/alter data in them like normal ustr's.

Note that although the length is specified manually above when ustr is running in debugging mode it will validate all strings, and will instantly spot a badly defined read-only ustr ... allowing you to fix the problem.

Solving the LICENSE problem

The License for the code is MIT, new-BSD, LGPL, etc. ... if you need another license to help compatibility, just ask for it. It's basically public domain, without all the legal problems for everyone that trying to make something public domain entails.

The "optimization" of string.h usage

With ustr, due to it keeping the data as a valid C-style string all the time, you can just call ustr_cstr(), to get a (const char *), or ustr_wstr(), to get a (char *), at any time. Use whatever algorithm you need to.

If you are worried about the "inefficiency" of growing the string to the correct size you can also pre-allocate a ustr of the size you think you need, thus using a single malloc() call (and memcpy()'s to add the data). This is basically as fast is it could be made to go by hand ... and it'll now be managed and safer.

One point worth highlighting here is that people often assume they know where the performance problems are going to be, however I've seen benchmark runs of C-style string using code where the function that used the most CPU time was strcmp(). For certain workloads general C-style strings are horribly inefficient at answering the question "are these two strings equal", ustr solves that problem with ustr_cmp_eq() and in a similar vein also provides ustr_fast_cmp() for when you need to sort the strings but don't care what order the sort is in.

GDB macros for easy debugging

The big problem here is that when you have a core dump you can't call any functions to find out data about the string. Thus. I've written some gdb macros here which mean all you have to do is type "pustr_all s1" etc.

A Significant example of usage, with comments

Ustr *s1 = USTR( "" ); Ustr *s2 = ustr_dup(s1); Ustr *s3 = ustr_dup_cstr( "" ); ustr_cmp_eq(s1, s2); ustr_cmp_eq(s1, s3); if (ustr_shared(s2)) ; if (ustr_ro(s2)) ; if (!ustr_add_fmt(&s2, "%s %d %c%d" , "x" , 4, 0 , 8)) ; if (ustr_owner(s1)) ; if (ustr_owner(s2)) ; foo_API(ustr_cstr(s1), ustr_len(s1)); foo_API(ustr_cstr(s2), ustr_len(s2)); s3 = ustr_dup(s2); if (ustr_owner(s2)) ; if (ustr_shared(s2)) ; ustr_free(s2); ustr_setf_share(s2); if (ustr_shared(s2)) ; if (ustr_ro(s2)) ; s3 = ustr_dup(s2); ustr_free(s2); ustr_free(s2); ustr_free(s2); ustr_free(s2); if (!ustr_add_cstr(&s3, "abcd" )) ; ustr_add_cstr(&s3, "1234" ); ustr_add_cstr(&s3, "xyz" ); if (ustr_enomem(s3)) ; ustr_setf_owner(s2); ustr_setf_owner(s1); ustr_sc_del(&s2); ustr_cmp_eq(s1, s2); s2 = USTR1(\x0b, "Hello world"); if (ustr_shared(s2)) ; if (ustr_ro(s2)) ;

You can look at/download the code here or the code for the examples here.