Planet Bongo

memmgr

2009-06-19T02:03:33+00:00

ok, ok. i get the hint. i’ll blog. everyone in the channel has been clammoring for more information and i’m terrible at sharing it. the startling lack of posts is not because there isn’t anything going on, but because i’m such a crappy writer, which you either already know or will soon find out.

a lot has changed since the last blog post. some of it i’m sure you are already aware of, but some you may not be. the biggest note regarding the trunk is that we’re in testing on the .5 release now. i took time to migrate my personal domain off hula and onto bongo which has been great. i can now find and fix the bugs that we were hitting on hula. there are still some hard to find bugs, but i’m betting that the other big news will help find it.

now on to the most amazing stuff…. the new memmgr branch. this has been a major blessing for the bongo project. back in the day, Novell engineers decided to write a custom memory manager for the product. i think it was back around the NetMail 3 days. i don’t recall it being there with NIMS, but i could be wrong. anyhow, it apparently greatly increased the speed of the product on Netware. it stuck around and we got the scrubbed version via the hula project. i recall in my programming classes in college writing my own memory manager and the benefits of them, however it does make some things a bit difficult. finding memory leaks and overwrites can be a bear. tools like valgrind had a very hard time with the memory allocation system as they would see one massive alloc and not a bunch of smaller ones.

i don’t know if there will be any sort of speed improvements over the custom memory allocator, however my gut says not really (note here that i’m no expert on memory subsystems…). the amazing benefit to us is that valgrind can sensibly operate with our binaries now. i just spent time debugging bongo-config on the branch and i’m surprised we even worked before given how things were coded. i was freeing stuff then using it and valgrind was useless to find it as the custom manager didn’t return the memory to the host, but kept it for pooled use later. this will be an incredible addition to our codebase.

another major change was cmake. alex wrote a bit about the make system on his blog, but i’d like to throw my two cents in. the new system is simply amazing and quiet. don’t get me wrong, there is nothing i like more than watching gcc compile endless .c files on a friday night (which is why i use gentoo), but this is awesome. percent complete and all. along with that is the quick installs and “make distclean” being as simple as “rm -fr {SRC}/build”.

the last major change, so far, with the branch is logging. we got rid of log4c and now just have a simple syslog interface. this will be swapped with something a bit more modular and configurable in the near future, but for now it works great. just be ready for spam in your syslogs (which is another of task on my todo — fixing the log levels).

overall .5 is incredibly stable on my production server, and the branch is now running on my test server. the store is running stable against my 1.4GB 151,400 (and growing) email mailbox. expect leak fixes and other improvements over the short term!

i did have one idea the other day based off a conversation with Chris at my work. an agent that can arbitrarily add, or change, header lines. this is the lowest priority task on my list as its use would be rare, but it might be nice to have. other tasks on the plate are the re-write of queue which we’ve been talking about for a while now. we have some good ideas on this. rewriting smtpd which apart from queue is the last major section of truly legacy code. and of course we can’t forget things like bongo-manager, bongo-queuetool, and my favorite, snmp.

there are a lot of tasks there, some required, some just dreams, but all i think good ones. this is starting to get fun!

A few words on CMake…

2009-06-16T22:47:26+00:00

I did promise Lance that I would blog more on Bongo, and I’m going to try to stick to a post a week at least - however, this first one will only tangentially be about Bongo.

Since the project was initially released, the autotools build system was what you needed to create Bongo. There are a variety of benefits to using autotools, and it’s an extremely well-tested and mature system. However, it’s also relatively difficult to understand and not particularly quick. Over time we accreted more and more things into our build which no-one understood fully and that would occasionally blow up in our face.

Now, a little while ago I started a new branch of Bongo, called “memmgr-glib”. This was mainly to replace the memory allocator - maybe Pat can blog a bit more about this, but the short story is that this change has highlighted a variety of bugs in Bongo. The branch version of Bongo therefore seems extremely unstable - actually, it’s just a lot less bug-tolerant, which is overall a good thing - but we used that as a cue to make some other changes.

One of which was a switch from autotools to CMake as our build system. Just to give one really obvious stat:

	autotools	cmake
Configure	52 seconds	3 seconds
Initial compile	47 seconds	21 seconds
Install	1m 15 seconds(!)	2 seconds

Clearly, there is a big difference in each category: it must be said that the build systems aren’t doing quite the same things at this point, and I haven’t done these tests properly as benchmarks, but fundamentally the times are extremely different. I expect eventually that the compile time will even out - the difference will become minimal, and is already of the same order. Configure and install are clearly quicker though, and there aren’t any amazing short-cuts being taken on the cmake side.

The huge difference this makes for developers is the compile-test cycle. A re-compile with either system is basically pretty quick - something like a few seconds each. However, “make install” with both systems is quite different. To do a proper update, it’s much quicker with cmake, which means the testing cycle is really quick.

One other thing that is also big, for me at least: with CMake, we can finally do out-of-source builds. It’s possible with autotools, but our system never quite got it right. With CMake, you can check out Bongo from svn and then do something like ‘mkdir build/; cd build/; cmake ../; make’. The source we checked out isn’t touched, and nothing gets built in the source tree - it all happens in another directory. Not only does it feel cleaner, but it means that you’re not likely to commit any files which shouldn’t be there and if something goes wrong you can just nuke the build directory and start again easily.

I’ve yet to teach the bongo-build bot on IRC about this new system, but I expect that it will make building there a lot quicker too - mainly because it goes through the cycle above more than once (it configures twice, for example). It will be interesting to see what a difference it makes!

Bongo Images and the 0.5 release

2009-06-15T23:19:44+00:00

Bongo Images and the 0.5 release

I have been working on the images for the 0.5 release of Bongo and I have now completed them.

They can be found here http://www.rpath.org/rbuilder/project/bongo/latestRelease

There is a change however in how we will be implementing the images from now on.

I have decided that to make it easier on you guys and also to limit the number of hours I have to work on them I will create a version 1.0 of the images which I will not change for the foreseeable future.

What will now happen is that we will just update the packages that apply to Bongo and release them for consumption.

I have also decided that I will start using the built in features of the rPath rbuilder system to start developing changes or updates to the bongo packages in the -devel branch and then when I am happy that they are ready for general consumption I will promote them to -qa which is an image I will be running permanently I will make this image available to anyone who would want to help test the bleeding edge stuff from Bongo.

Once the -qa has been completed I will promote to the release branch which will be bongo.rpath.org@rpl:bongo-1.0.

This will automatically make the changes available to you guys and so we will become a rolling update project. So you will never have to re-install your server again unless you have to upgrade your hardware.

I hope you find it easier to use. We are now well on our way to making Bongo a great mail server.

Bongo Images for 0.4.0 have been fixed

2009-06-10T19:14:33+00:00

I have been able to rework the Bongo images to make them work.

I would appreciate any testing that could be done as I do not have all the vm technologies to test.

So I hope you have fun .

Here is a link to the downloads

Bongo Images for 0.4.0 Broken

2009-06-03T19:20:00+00:00

Due to a server error where the images get built an error has crept in that make the images useless.

This error has been prevalent for some time and so it has caused the images to break.

I have therefore pulled the images and will try to fix the problem as soon as I can.

I will post again when things get better