We were given another chance to meet in Extremadura to discuss vcs-pkg issues, after the first opportunity was too short notice.

Currently, the tentatively scheduled dates are 2-7 September 2008. You can get the details from the wiki page. If you’re interested, please reserve those dates and add yourself to the list of participants.

NP: Hooverphonic: The Magnificent Tree

Looking at the current discussions, I'm wishing some people would calm down a bit. It's always impressive how some things switch sides like pendulums.

First of all, Debian already is centered about packing software and not developing them. We already have the rules and policies and methods in place. Our policy states:

If changes to the source code are made that are not specific to the needs of the Debian system, they should be sent to the upstream authors in whatever form they prefer so as to be included in the upstream version of the package.

And our source format shows how important marking the difference is to use: We have explicit .diff.gz files to contain them. The differences are not hidden in same Version Control System (like BSD) or in propretary formats (ever tried to unpack a .srpm without rpm or without downloading some magic perl script?), but in a simple universal understandable format.

That said, please remember we are a distribution. Our priorities are our users and not the whims of software authors. We have to find the middle ground between harmful and necessary changes. Patching software to abide the FHS, to allow the user choose their editor or browser in an common way or any other thing to form a coherent set of packages is no bug in Debian, it is a bug in upstream to not allow this at least via some configure option. We have neither the manpower nor the job to rewrite and fork stuff to a usable state, though. Thus we have to keep to upstream and hope they will include our modifications or forward-port them to every new release.

Thus, we need both: We need to patch (and in general, the worse the software from a distribution's point of view or from a general point, the more of this we need. This does not necessarily hold in the other direction, though.). And we need and want to show what we change. So our users can find out how exactly the software we ship is different. And so other people dealing with the same software can profit from our changes. (After all, free software is about "giving back" a lot.). And of course maintainer change over time and we want the new one understands what the previous one did.

That said, there is of course things that can be improved. But as with all improvements there are things that improve and there are things where nothing is worse than good intentions.

Adding more thing to keep in sync is almost always a bad thing in the long run. The easiest way to keep things accessible is to use stuff actually used. I think any additional place to track patches will be futile. A good interface to view the .diff.gz files in our archive in some browser, on the other hand, could hardly get unuseful.

The format of a single .diff.gz is of course also improveable. Things like a quilt-like standardized patch series look like a very good idea to me. But the format is of course also very downgradeable. Storing VCS formated information for example, tempting as it is, spoils two very important points:

• universal readability is lost. Everyone coping with software can work with patches or patch series, no matter what tools they use.
• visibility of changes is sacrificed for history. Every single VCS I ever saw primarily coped with history. Even patch based ones represent changes in time and not by itself changes by topic.

And of course, a lot can be improved by just using some rules more strictly. Not using a pristine tarball or a proper subset of one where the first is not possible should not be some mere ignorable non-conformist behaviour, but seen as the serious problem it is.

I’ve just released pommed v1.18, which is mainly a maintenance release.

I’ve added some more USB IDs for Apple external keyboards and relaxed the event devices identification for internal keyboards.

pommed will now happily start on a machine fitted with a keyboard+trackpad assembly that normally isn’t found on this model. This can happen when the topcase of the machine is replaced with the wrong part. I expect this situation will happen more and more often as people buy parts from EBay or similar to repair their laptops.

As I said a month ago, my main server Delmak was dying. Well it still runs (proof: you could read this blog some days ago).

Thanks to friends I host for free, they've kindly given enough money to buy a brand new server (C2D E8400, 4 GB RAM, 2x500 GB RAID 1) in order to replace the good old Delmak. This new box has been named Prometheus after the only BC-303 class battleship ever built.

Delmak was used to mainly run as a Web, mail and databases server. I decided to do use this server switch to change the server software I use.

The first mail server I setup was based on Exim 3, courier-{imap,pop}{-ssl,} with userdb files. That was... rough. Later I switched to Exim 4, using vexim, and MySQL as a back-end. That was something like 3 years ago I guess. Since then I never really touched that back. I added spamassassin and clamav filtering some months after, because some users asked for it. That's all.

So this week, I decided to switch away from this configuration. I do not understand Exim anymore anyway, so I decided to use Postfix which I often use and administrate at work. Obviously, I also now use PostgreSQL as database back-end, since it rocks, and since Postfixadmin supports it. By the way, be aware that the Debian package of postfixadmin is crappy (the configuration file is readable by anyone by default, with the database password in it). I also set up postgrey which is quite nice and efficient.

Well, then was time for amavisd-new installation, but I did not do it. Seriously, amavisd-new configuration is a bloody mess, as the language it is written in (yes, Perl).

So I switched to dspam which I heard is nice. Well, it seems to be for now, since it even supports clamav daemon usage directly, which is very very nice because that means I do not have to set up another thing for that.

I also switched from courier to dovecot, mainly because the latter seems to be faster and lighter. I then changed the default virtual_transport to Dovecot LDA. The main advantage of this is that the LDA updates the Dovecot index while delivering. It also supports quota, which I do not use and plug-ins, like the Sieve language for mail filtering.

So I decided to change my procmailrc to a new Sieve filter. My procmailrc is quite small since I only use regex to match lists and some mail address, so it has only something like 12 rules. And well, I did not do it since I discovered after some googling that Dovecot implementation of Sieve is grabbed from Cyrus which does not support variables for now. That means that the following procmailrc code:

:0:
* ^X-Mailing-List: <debian-.+@lists.debian.org>
* ^X-Mailing-List: <debian-\/[^@]+
list-debian-$MATCH/

which will translate to:

require [ "regex", "variables", "fileinto" ]
if header :regex "X-Mailing-List" "<debian-(.+)@"
{
    fileinto "lists.debian.${1}";
    stop;
}

But that won't work since Dovecot Sieve implementation does not support "variables". Well, since I'm not ready to list all the lists I'm subscribed to, Sieve is a no-go for now. I'll stick with procmail.

(tl;dr: if you have a few minutes, please add information here)

Joss,

the problem with the new package formats is that there is nothing that actually uses the additional information in a way that adds significant new functionality, so the net result of the change was that we throw away the information at a different layer in our software stack, and one of the interfaces got a lot more complicated in the process.

One possible application would be a "poor man's patch tracking" inside the BTS, perhaps with a new state "fixed in patch".

I can see two ways of implementing that:

1. by extending the interface of the "new" package formats that {Debian,Ubuntu} bug numbers are attached to the actual patch files and having the archive maintenance software extract and process that information (reject packages that add a patch for a bug without closing it in the changelog, notify the BTS), or
2. by leaving the package format untouched and simply adding a regex matching "Fixes: #nnnnnn" that is reported to the BTS as "we have added a patch", so the submitter is notified that the bug is gone for him/her; the bug is then closed in the changelog of the upload removing the patch.

The former approach also allows us to link to patches from BTS pages, which the latter doesn't, so there could be actual benefit here if we believe it is worth the additional complexity.

(Update: Raphaël thinks it is. I like the idea of a package format with separate patches a lot more in this context than I did without it, but still my fear that it will actually be perceived as sanctioning large patchsets still remains.)

About mandatory co-maintenance: the problem isn't "helping". We have plenty of people with commit access to packages they don't even remotely understand who are really helpful (not). The problem is that someone needs to actually read all the commit logs and understand what the changes do in this context. In most cases, that person or group would be upstream, not a DD.

My first impression after reading the patch was "adding uninitialized data to the entropy pool is pointless/harmful as it is not random, so this patch makes sense", because the loop around it was not contained in the patch. Obviously I'm not an OpenSSL developer.

There is nothing Debian could have done internally to verify the correctness of this patch that would properly scale to the entire archive, even if we put "more emphasis on security". The only solution I see is reporting every patch to upstream immediately and getting affirmation that it is correct.

This, however means that we need to produce patches that upstream can accept. For obvious code bugs, that is simple, but for integration patches like paths it is not sufficient to replace one string with another, but rather make it configurable in some place that can be reached from debian/rules.

In an ideal world, we end up with very few Debian specific patches, so essentially we are talking about adding functionality to dpkg that we don't want to use.

I've started a page in the Debian Wiki, Getting Packaged with an outline of a possible document aimed at upstream developers that should list the typical problems we run into and how to avoid them.

gnupg (1.4.6-3) unstable; urgency=low

* Adopt package. Thanks to James Troup for his work in the far past.
Thanks to NMU'ers Bastian and Thijs. (Closes: #476418)
* Co-maintainers wanted.
* Don't build-dep on pcap on non-linux-archs. (Closes: #357267)

-- Sune Vuorela Sat, 17 May 2008 15:42:55 +0200

TODO:

• Coordinate with release-team and d-i what changes can still be done before lenny
• Look at the new upstream version
• Get a alioth project and host stuff in a version control system
• Decide on version control system
• Get a team and figure out team workflow
• Look at this newfangled packaging thing called debhelper
• Make all bugs either closed or forwarded

So - anyone interested?

You might remember that I wrote a blog entry about my switch to the green electric utility "Lichblick" (Germany) a while ago. I did that purely out of environmental reasons, I didn't want to continue to waste money on polluting and/or dangerous crap such as fossil or nuclear power. Yes, even if that meant a slightly higher price (but I really didn't compare prices much before switching — I was after an environmentally clean solution, not the cheapest solution).

Quick status update: the switch went really nice and easy, no downtimes, no hassle. I've been a happy customer for more than 8 months now.

Today in my snail mail inbox: a letter from Lichtblick that they're going to reduce the price per kWh from 20.25 to 19.99 (Euro) cents starting July 1st and they give you a guarantee that there won't be any price raises before the end of 2009 (more details also here). Now, that's a positive surprise there.

Compare that to 98% of all other energy providers in Germany who have lately increased prices quite a lot for very obscure or non-existant reasons.

Yes, I do realize that the reduced costs are not that dramatic, and Lichtblick is using this as a means to impress people and gain new customers. But I fully support them in doing so, the more people are switching to a green energy provider the better, if you ask me. I encourage everyone to consider switching, either to Lichtblick, or some of their competitors (in Germany) e.g. Greenpeace energy, Elektrizitätswerke Schönau, or Naturstrom AG. There are various alternatives in other countries too, of course.

There are obviously some things we need to remind if we don’t want something like the OpenSSL debacle to happen again. It doesn’t mean we need to throw stones nor to rush into changing our processes without thinking. However, there are already some things that should be obvious but unfortunately are not.

1. Shipping a giant diff.gz that contains all changes in one, putting security fixes, policy fixes, bug fixes, cosmetic changes and autotools files at the same level, is not something we should accept anymore. Improvements in the dpkg-source format are much welcome in this direction, but they are useless if maintainers don’t use them. Neither a VCS nor a build tool will be able to know which line of the changes is related to which bug. Only the maintainer can.
2. Core packages should all have co-maintainers. This is pretty much stating the obvious, and is much easier said than done. The OpenSSL case is one of the best examples here: Kurt is not one of those who refuses help, but frankly, would you want to maintain that package? Having already maintained packages with messy code, upstreams not understanding at all the needs of a distributor, avalanches of security alerts and randomly-changing ABIs, I can tell you this is no fun like it can be to hack on a desktop environment or a device driver. The only sane reason to do this is that you need the package to work. The only visible result you get from your work is that programs are not randomly crashing.
   I have no magic recipe to propose so that more people help with such packages, and that’s where we need to be really innovative. Cross-distribution teams, mandatory co-maintainership on a core package for each DD… these (and all ideas I have not heard of) are the experiments we should start now.
3. Patching bad code leads to unpredictable results. What maintainer of a complex package has not introduced a new bug while trying to fix another one? Even when a piece of code is maintained by uncooperative developers, is not commented, uses arcane variable names or is impossible to understand without having contributed 3 winning entries to the IOCCC, it needs to evolve. And in these cases, it is only a matter of time until such things happen.
   Don’t get me wrong: I’m not trying to put the blame on upstream here. They have contributed very valuable code to the community and their work helped in the considerable widespread of cryptography. It’s just that their code is not enough for our needs. If we can’t patch it safely (and I’m now convinced we can’t), maybe we need to focus on alternatives and help them getting used by crypto-related packages. The code in GnuTLS and NSS is not necessarily better, but most (if not all) patches Debian needs to apply to them are build and portability fixes.
4. Unless Debian-specific, 1 patch = 1 bug in the upstream tracker. This should be obvious, but given the number of patches that are never forwarded, it doesn’t seem so. You should not only give a chance for upstreams to review the patch, but you need them to track it, and you must give them the chance to review it anytime someone else stomps on a similar issue. If upstream does not have a bug tracker, they probably think their software has no bugs. Which means they are not trustworthy, and we go back to point 3.
5. We need to give more priority to security. Issues in the security team seem now fixed for good and they have been doing an awesome work. There isn’t much left to do so that packages are all built with security-hardening features, but it still needs to be done. And there is much more to do so that we can provide out of the box a decent SELinux setup, or, if it turns out unrealistic to do, a decent system hardening setup using another framework. I know the SELinux zealots will jump on their high horses to explain that their framework is better, but the current situation where it is impossible for the average system engineer to setup a Debian-based MAC system is much worse than having a suboptimal setup that already works.

All in all, this incident has a great impact on Debian’s image. If we don’t react accordingly, adapting our processes and our system to match what our users expect from us – and they expect the best – they will turn away from us. With very good reasons to do so.

Update : It seems OpenSSL does have a bug tracker. Thanks Kurt for pointing me to it.