pkgsrc and the concepts of package management 1997-2007 (part 2)

Welcome to the second, concluding part of the "10 years of pkgsrc (1997-2007)" interviews (the first part). This issue provides more talks with developers and users of pkgsrc, alternative concepts and different systems are represented as well.

I wish to express my gratitude to all participants for the cooperation and the great interviews!

As always, feedback, corrections and additions are welcome, just contact me via email.

Please note that the expressed opinions and the selection of topics do not represent the official views and directions of the pkgsrc project.

- Mark Weinem


pkgsrc and alternative ways of package management


Developer Roland Illig about pkglint and the pkgsrc documentation

Hi Roland, please introduce yourself: 

I'm Roland Illig, a 27 years old student of computer science. I live in Hamburg, Germany and besides hacking on pkgsrc, I like to play Go and other strategy games.

My first contact with NetBSD was in 2002, when I installed it on an old machine. Back then, NetBSD was at version 1.6, and there were 6 CD-ROMs with all the packages on it. It was easy to install, until I needed to update a package because of a vulnerability. At that point, I started to use pkgsrc. First I was scared by its complexity, but slowly I got accustomed to it. Because I had often heard that NetBSD had "high quality code", I thought that would also apply to pkgsrc, and I dived into it. I met lots of ugly, bad code, and I decided to improve that situation. I started by hacking on pkglint, a tool that checks packages for common mistakes.

I joined the pkgsrc team in 2005. Since then, I learned much about the internals of pkgsrc and also worked on the pkgsrc infrastructure. One of the things I didn't like about pkgsrc was that the code had almost no error checking, and my work on both pkglint and the pkgsrc infrastructure was mainly to change that, so that pkgsrc gets more reliable in case of unforeseen situations.

Where and how do you use pkgsrc? 

There are three places where I use pkgsrc: At home, on my desktop machine (NetBSD), at my university (Solaris) and on a web server (Linux). At home, I am running a bulk build to have all the packages available whenever I need them and to find those that don't build. At the university, I am using pkgsrc to provide me and my fellow students with "the usual software" on Solaris. The staff at the computing center does not have all the software installed, so I provide an additional set of software packages. On the web server, there is a preinstalled Linux distribution (SuSE), but the thing is that package updates are not as easily available as I want them. Because I know pkgsrc, I am using it there, and it works pretty well. It is also easy to manage local patches for the packages and to configure them with non-standard options that are not available in the usual distributions.

I also don't like the policy of Linux distributions to add lots of patches to existing software, so I'm happy to have the original software running, managed by pkgsrc. At all three places, I am using pkgsrc as an unprivileged user, because I don't like the idea of running arbitrary software as root, even if it's Open Source.

What are the pkgsrc tasks you are currently working on? 

I don't really have tasks that I work on. I mostly build packages or look at the code in the infrastructure, and if I find something that looks like it should be improved, I try to do that. From time to time, I work on things that later become bigger and bigger, like the online documentation (bmake help). It started as a small program to query the existing documentation, and since I found it useful and there are still many undocumented parts of pkgsrc, I am slowly adding new documentation whenever I need it.

In the first two years of my pkgsrc career, pkglint was my main focus. It has matured a lot in that time, and I only modify it to adjust it to the changes in the infrastructure. The many warnings that are printed by pkglint give me an idea of the infrastructure parts that are inconsistent or can otherwise be improved, and I try to fix those things so that the resulting code "feels good".

At the moment, I'm improving the pkgsrc support on Solaris, to get more packages usable in the university. I didn't spend much time in the last few months because I'm heading towards my diploma thesis, and I'm distracted by playing Go and other games. :)

Can you tell us more about pkglint. What is it useful for and how shall it be used? 

The pkgsrc packages are mostly written in the form of Makefiles. When writing this kind of files, you have lots of syntactic and semantic freedom that isn't caught by bmake itself, and in many cases, it's entirely impossible for bmake to check anything about the contents of the variables in a Makefile. This situation is similar to the one of the early C compilers which didn't have function prototypes. One could write almost anything, and it would be interpreted somehow by the compiler. There are many cases where developers write code that is syntactically correct but misses important things like error checking or quoting variables (in shell scripts, or in make(1) macros). I have taught pkglint to check for these things and to inform the developers about them, so that they can concentrate on making other, more interesting, mistakes. There are many conventions on how package definitions should be written, but most of them are not documented anywhere. This is where pkglint steps in, providing the documentation in form of code, so you don't even have to read documentation to see what the conventions are.

The second part of the question concerns when pkglint should be used. I would say, before committing a non-trivial change to a package, you should run pkglint on the package, to see whether you have made one of the typical mistakes. If there are no warnings, you may try again with the -Wall option, which also warns about things that are merely aesthetical or that result in too many false positive warnings. And if you have questions about the warnings, please first ask pkglint, and if it doesn't have an answer, ask me. Using pkglint is kind of an automated peer review. But bear in mind that I have used pkglint a few times to enforce my personal view on how packages should look like. (In some of these cases, I have reverted the changes after I realized that I were wrong.) Not everything pkglint warns about should be fixed without thinking, so don't switch off your brain when using pkglint.

pkglint is available via pkgsrc (pkgtools/pkglint). See the pkglint man page for details about the use of pkglint.

Please tell us about the state of the pkgsrc documentation: 

When I joined the pkgsrc team, most of the documentation had been in the developer's head, but it was not publically accessible or even in written form. At least, I had this feeling when I was a pkgsrc newbie. Especially the implementation of the pkgsrc infrastructure was hard to understand unless you asked the developer who wrote the code in question. In the last few years, the situation has changed a lot. Many of the files of the pkgsrc infrastructure are documented at the top of the file. (Well, except for those parts that are hard to understand, like buildlink3, the tools framework and the compiler wrappers.) I usually accessed the documentation using grep -wr $keyword $pkgsrcdir/mk Since that is a lot of typing, I wrote a little program that does that work and also formats the output. This program is run by typing bmake help, which is a mixture of the man(1) and apropos(1) commands. I am constantly adding keywords to the help files, and my vision is that a pkgsrc user or developer can just type bmake help topic=whatever to get useful documentation about whatever this "whatever" is, be it a variable, a make target, a C function that occured in an error message of an "undefined reference", or anything else.

The pkgsrc guide has always been a good source of documentation, and it would probably be best to merge the short reference documenation from "bmake help" into it, so that you only have to look at a single place. In general, I think the state of the documentation can still be improved in large parts. We are missing simple tutorials for doing all kinds of work, and many of the recently added pkgsrc features like cross-building are not yet documented in an introductory document. pkgsrc is such a powerful and flexible system, it's just that people don't know about all these features if we don't document them.

One thing that I find important for the documentation is that the things are explained as easily as possible. That means that pkgsrc must be well-designed so that the documentation can reflect that, and numerous times I kept off from writing documentation because I didn't have the feeling that pkgsrc solved some problems in the best way possible. I just don't find it acceptable to document a bad implementation when you can have a good one instead.

The NetBSD wiki has been a recent addition, and it has proven quite useful, at least for me. In the pkgsrc category, there are some important documents for things that I had never found documented before. For example, "The pkgsrc portability guide" documents all the quirks that one must keep in mind when writing portable shell scripts or C code, and "Typical pkgsrc error messages" try to provide a generic help with all kinds of problems that occur when building packages. So in conclusion, there is plenty of documentation, including the mailing lists, but it is not yet organized in a way that I consider high-quality.

The pkgsrc guide is available online and for download as PostScript and PDF files.

If you analyse the current state of pkgsrc, which improvements and changes do you wish for the future? 

The thing that I am interested most at the moment is good support for binary packages. Compared to typical Linux distributions we are lacking a lot of comfort here. I'd like to see a well-designed API for working with binary packages (libpkgsrc), at all levels: manipulating installed packages, installing, deinstalling and updating packages, managing sources for binary packages. I'd like the tools for the binary packages to be more robust: When I interrupt one of the pkg_add or pkg_delete tools, I want the package database to be in a consistent state.

I would also like to know the exact state of the files in /usr/pkg (or, more generally, LOCALBASE): Which files are orphaned, which ones belong to more than one package (shouldn't happen), to which file does a package belong (pkg_info -Fe doesn't work in many cases), what changes have been done to the configuration files? Another high-level goal of mine is to make pkgsrc self-explaining. That means that users can query all the details of pkgsrc (bmake show-all) to see what is happening behind the scenes. Building software packages is a complicated process, so we should help any interested persons to get used to pkgsrc easily, which as few secrets as possible. The binary packages that are generated by pkgsrc are not yet checked to conform to any conventions. For example, on Debian, there is deblint, which checks that the binary packages conform to all kinds of useful conventions. Currently, we don't even have a document that summarizes what a pkgsrc binary package looks like, what it contains, and what it doesn't.

Compared to its beginning 10 years ago, pkgsrc has improved a lot in terms of portability, cleanliness and documentation. But we are still away from having a perfect package management system. I even think we are not yet "good" in some areas, but nevertheless, pkgsrc is usable enough for me that I don't want to miss it.

What is your vision for the crossbuilding of packages? It is possible to achieve as specialized Linux-based build systems show. Why doesn't it work as well with pkgsrc? 

My vision is that we are able to build packages for any supported platform on any other, for example, building NetBSD-4.0-vax packages on SunOS-5.10-sparc and vice versa. Should this ever become possible, I am confident that we have understood cross-building correctly. Currently, I have the feeling that it's more a hack than real cross-building support, since our dependency system does not feel like it would support cross-compiling: We don't have a way to explicitly say: I need a native awk for building the package, and when the package will be installed, the awk from /usr/bin should be used. For many Perl modules, we will need a native Perl installed (for running Makefile.PL), and we also need Perl for the target platform to be installed (for letting Makefile.PL check if the package dependencies are installed). This cannot yet be specified in an easy way.

pkgsrc has not been designed with cross-compilation in mind, so now, that we feel a need for it, we must add it to the existing system. It interacts with many of the other complex parts of pkgsrc (buildlink3, wrapper, tools, DESTDIR support), and getting all this right needs much knowledge, so I expect cross-building to take a while before it becomes usable. There has already been some work by Krister Walfridsson and lately Joerg Sonnenberger. I didn't follow closely what they did, and I must admit that I didn't even try it out.

I think pkgsrc is able to cross-build packages the really hard way, that is from one platform to a completely other (operating system, architecture). Most package management systems don't even support that many platforms, so when it finally works it will be really cool.

Are there things you like about other package building and management systems? Any concepts you would like to see integrated into pkgsrc? 

I like Debian's deblint, but haven't found the time yet to make a similar tool for pkgsrc. I find the *-debuginfo packages from SuSE Linux impressing. They install the source code of a package together with debugging information, so that you have it available when you debug programs. Most Linux distributions have a graphical tool for doing binary package management, which would also be a nice addition to pkgsrc.

pkgsrc and all other package mangement systems seem to have a bias towards continuous growth. The concept of downsizing doesn't seem to be included. Fewer and selected packages to provide more quality, stability and full functionality - should pkgsrc have a stricter focus? 

I don't think pkgsrc's focus is too strict. pkgsrc is used in a lot of different places, and we do not know which packages are needed in each case. Of course, if there are some packages that had been useful five or ten years ago, but which nobody uses anymore, it would be good to get rid of them. But we currently don't have a way to find out. There is the pkgsurvey package that reports the list of installed packages back to us, but we cannot draw the conclusion that any package not reported by pkgsurvey is unused.


Developer Ulrich Habel about pkgsrc on Solaris

Hi Ulrich, please introduce yourself: 

I live in Munich, Southern Germany. I work as a sysadmin for a local broadcast company. I started to explore the worlds of NetBSD on an Apple G3 powerbook in 2000 after I got bored by MachTen. I use NetBSD on my laptop and on my workstation at work. I feel pretty comfortable with it. When I am not fighting NetBSD things or updating my packages I enjoy cycling, geocaching or sailing.

Where and how do you use pkgsrc? 

I am using pkgsrc on all my systems as it turned out to be quite flexible to all my needs. I am using Solaris 9, Debian and NetBSD. I am way too lazy to learn 3 package managers, learn how to package 3 different parts from one software which needs to be deployed on all the hosts.

At work I am using pkgsrc on Solaris 9 for 3rd party software. It started with very few packages like openssh, openssl and some perl modules. After some time I maintained about 30 packages and looked for a build system. At this time pkgsrc seemed to be the best solution.

Several package management systems are available for Solaris. What do you like about pkgsrc and why do prefer it on Solaris systems? 

First of all - because it works. Solaris has a really weird userland and you are always missing the basic tools. There is a SUN Compagnion CD which is assembled by SUN, however I am quite sure about updates. If I need to package my own packages I still have to build my toolchain myself. pkgsrc is a toolchain for building, installing and adminstrate software. Administration is quite simple - however, the update process of packages needs to be improved.

What was your motivation to become a pkgsrc developer and what are your development goals? 

I ran a small website located at http://sunpkg.de where I uploaded binary packages for Solaris based on my bulk builds. I made a bootstrap package for Solaris in the native SUNpkg format to give admins an easier start with pkgsrc. There is no particular reason why I wanted to join the NetBSD development team. After several years of contribution I wanted to be part of the team.

My development focus on NetBSD pkgsrc are the Perl packages. I'll take more and more of them as a maintainer. I really would like to see NetBSD as premier platfrom for Perl related things. Right now we have a lot of Perl modules packaged already, let's keep them updated.

What are you currently working on? 

Right now I am working on a Perl script which analyzes the Perl module related Makefiles and keep them updated according to CPAN automatically. This sounds quite simple, however there are few things to deal with and it will take a serious amount of time.

Which improvements would you like to see in pkgsrc? 

We need a more flexible way to keep things updated. It's just a waste of time to update a NetBSD pkg system. There had been the term "NetBSD pkg Mikado" around for a few years and we need to get rid of it. I would like to see a more comfortable way to get notified about available updates for our packages. Maybe a NetBSD jabber server which a certain pubsub service might be an approach to the problem. This is just an idea.

Another thing which should be talked about is the release cycle of the pkgsrc branches. Quartly releases are a very nice thing for endusers with workstations only. If you run a server farm , however, you won't be able to update them that often - except for security related updates, of course. I can't imagine a reason why updating is required in such a short period of time. If someone needs current packages they can move to -current pkgsrc which is bleeding edge, sometimes it's even bleating. pkgsrc-wip is always in a developing state and works quite well.

Are there any topics left you would like to talk about? 

Let's make NetBSD and pkgsrc more user-friendly. The technology is great most of the things are working out of the box, however a few small things are just annoying. If I install GDM as my primary login screen, all the window managers should be listed there. We have a problem with man-power, however it might be easier to get more people working on pkgsrc if it's easier to use.


Developer Thomas Klausner about pkgsrc-wip and the updating of packages

Hi Thomas, please introduce yourself: 

I'm an Austrian in my thirties and live and work near Vienna. I've studied mathematics and am now working in a software company

Where and how do you use pkgsrc? 

I'm using pkgsrc on NetBSD on my home machines and some servers for which I am system administrator.

You initiated and maintain the pkgsrc-wip project. Please tell us more about pkgsrc-wip

pkgsrc-wip is a project for making it easier for non-NetBSD developers to get involved in pkgsrc. Everyone can get commit access there and commit and improve their packages. Some active NetBSD developers also give feedback, so you can improve :) It is also a place to work on development or unstable packages; currently, they are the testing ground for the new KDE4 packages.

Are there any other pkgsrc projects that you are currently working on? 

I'm an active member of pkg-bug-handler. This is a group that takes care to assign new pkg bug reports to the people who are most likely to fix them, usually the maintainers. Sometimes we just commit the patches ourselves, as time permits.

I'm also a member of pkgsrc-pmc, a commmittee which oversees the general development of pkgsrc and decides technical questions if there are disputes.

Considering that a simple make install is often not successful, is pkgsrc too heavy to use for ordinary users? 

Stable branches are a step in the direction to make it easier for ordinary users -- packages don't change that often there, and thus the regular breakage (introduced by library updates or infrastructure changes) is much reduced.

It would probably be helpful to get more developers interested in working on the stable branches to make them even more useful, but as developer time is limited, I actually prefer the current handling.

Additionally, the NetBSD Foundation is working on regularly providing binary packages for a few platforms -- I think the availability of binary packages will be a huge improvement for the casual user who just wants to run programs.

With pkgsrc there is no easy or faultless way to upgrade all installed packages. The upgrading procedure is at least bumpy and time-consuming. And are there possible solutions? 

I'm an advocate of bulk building packages in a sandbox and then installing binary packages on the work machines. That way, even if 'make install' doesn't work, you can still use the existing packages. Look at mksandbox from the pkgtools/mksandbox package for creating a sandbox, or use the pkgtools/pkg_comp package, which tries to do most of the work for you.

For updating the installed packages, I use pkgtools/pkg_chk, but pkgtools/pkg_rolling-replace looks very useful as well.

At the pkgsrcCon 2006 in Paris, you have led a roundtable discussion about the updating of packages. What was discussed and what were the results? And has anything changed about the updating problematic since then? 

We discussed various ways of updating packages (make update, make replace, bulk builds, using binary packages, ...). Since then, some tools were added to pkgsrc that make updating easier (like pkgtools/pkg_rolling-replace), but IMO we haven't found the definitive solution yet.

Are there things you like about other package building and management systems? Any concepts you would like to see integrated into pkgsrc? 

It would probably be good to add some GUI tools for finding and selecting packages to install, but pkgsrc.se does a good job of presenting the available programs in pkgsrc already.

I must admit I don't have much experience with other package management systems, except FreeBSD ports (in which I didn't find any features that I feel pkgsrc is missing).

Do you have any practical tips to share with the pkgsrc users? 

Use a sandbox with a bulk build -- it's worth the time to set it up if you're building the packages yourself. [Article: "How I learned to stop worrying and build pkgsrc in a chroot"]

If you have extra time, take a look at pkg_chk -N it will tell you about packages you have installed for which newer versions are available upstream.


Getting started with pkgsrc: about pkgsrc-wip and pkgsrc Hackathons

Adam Hoka and Peter Bex are pkgsrc users and maintainers of various software packages in pkgsrc-wip. They also initiate pkgsrc Hackathons since November 2007 (NetBSD's Hackathon page)

Please introduce yourself: 

I'm Peter Bex.

I'm Adam Hoka, 21 years old NetBSD user from Hungary with a passion for Unix. Though I'm not a long time NetBSD user, I'm trying to be a useful member of the community. I'm currently seeking an IT career.

Where and how do you use pkgsrc? 

Peter: I use it on all my own systems (desktop at work, laptop and server at home). Of course they all run NetBSD!

Since I started using pkgsrc I've always built from source, mostly because binary packages are not very up-to-date or well maintained, but also because it allows me to finetune packages with the options framework.

After I discovered a great tutorial on how to use pkg_comp (link) I've used it exclusively on my systems. It's great when you're developing packages; it keeps your main system clean and you don't have to worry about messing anything up. It's useful for users too, if they're using pkgsrc-CURRENT because that can break occasionally.

Adam: I have first used pkgsrc on Linux after a friend told me about it, but it has only became my primary package management system after I've switched to NetBSD and fell in love with both of them. :) I have used (or still using) pkgsrc on other architectures too, namely FreeBSD, SunOS, OSF/1 and NT's Interix subsystem.

What do you like about the pkgsrc system? 

Peter: What I like most about it is that it's very user-friendly, especially considering that it compiles everything from source.

For beginners it's enough to download and extract the tarball and cd category/package && make install Also important is that you have everything in one place, which makes installing a package very comfortable; you can just browse the categories and if a DESCR sounds like what you need, you can install it and try it out.

Another thing I like about the pkgsrc system is that runs about anywhere, just like NetBSD itself :) I ran Mac OS X for a while on my laptop, and used pkgsrc to get the Unix programs I needed. No need to learn an OS-specific package manager; I could just continue to use what I was already familiar with. I assume that sysadmins who manage large heterogeneous networks will benefit from this even more.

Adam: It has a very nice abstraction which makes it flexible and platform independent. I can use it on any other system which lacks the software I would like to use. The 'unprivileged' mode is very useful if I need to install and maintain software without root permissions.

I also see some very nice progression in pkgsrc's evolution, especially the recent works of Joerg Sonnenberger and Roland Illig. I also like that whenever I contacted a package maintainer for some reason, they were always very helpful and responsive.

What is pkgsrc/wip and what is special about it? 

Peter: pkgsrc-wip is a community project on sourceforge. Literally anyone with a sourceforge account can sign up for commit access.

Before wip existed, users who wanted to import a new package would have to create the package, make it available on the net somewhere, send a PR and hope someone would pick it up. If things needed improvement or a package needed updating, they would have to go through this entire process again, until they got fed up with it or got commit rights :)

It's important that not just anybody can have commit access to the main pkgsrc to ensure the quality of packages, but it's also important to have an easy way to become a package maintainer. pkgsrc-wip provides just this. It's also a great testing ground for experimental features like building packages straight from CVS or svn.

Adam: pkgsrc-wip is a place where one can experiment with pkgsrc, learn to make packages and maintain them there. It's also a testing ground for very experimental packages which would be too unstable for upstream pkgsrc, or you just want to maintain independently of upstream pkgsrc.

What was your motivation to become pkgsrc-wip maintainers? 

Peter: It was, like most things in the F/OSS world, a case of "scratch your own itch". I simply needed a package which wasn't in pkgsrc yet, so I created it in wip. I can't remember which one it was, but after that I created other packages, some of which have ended up in pkgsrc.

Adam: When modular Xorg was introduced to pkgsrc, some packages weren't supporting it yet. Xfce 4.4 was one of them, but I wanted to use it with modular, so I decided to add support to it. I enjoyed working on the packages it and I still do. I have always wanted to contribute back, so I took the chance and sent patches.

With the aid and hints of experienced pkgsrc developers I could learn a lot of things about pkgsrc's inner workings. Well, I'm not an official NetBSD maintainer yet, but really hope I can become a member of the project soon. :)

You initiated a pkgsrc Hackathon last November. It was the first Hackathon that was not directly coordinated by NetBSD or pkgsrc developers. Please tell us more about this "pkgthon", the goals, the results and about the experiences you made: 

Peter: We noticed there were many open PRs for pkgsrc, a lot of which were outdated. Also, the options framework had been around for a while but a lot of packages either didn't use it or rolled their own options system. The ones that didn't use it were most important, since adding options made it possible to compile them against fewer dependencies, which makes it easier to keep out bloat from your system.

Previously I had helped out a little with the Docathon and I lurked in #netbsd-code [IRC channel #netbsd-code on irc.freenode.net] at another hackathon both of which were lots of fun, so I decided to free up some time to focus on the hackathon. At the time it felt to me like progress was slow, but in retrospect we got quite a lot done. It felt slow because many of the important packages are pretty complex pieces of sofware which are hard to package.

Adam: One of the aims of this pkgthon was to improve option support of pkgsrc packages, especially at the field of multimedia support. And of course we could close a lot of PRs (special thanks to Aleksej Saushev), which was in fact more important than the options part.

The lesson learned for me of this Hackathon was that people only need a little motivation to achieve very nice things. NetBSD and thus pkgsrc have a relatively small, but very strong and helpful community. And it was also really fun so I recommend everyone to come to the next one.

Will more pkgsrc Hackathons follow? 

Peter: Absolutely! One of the things I have in mind would be a wip hackathon to remove or update some outdated packages and test packages so they can be moved to the regular pkgsrc tree. To make this happen we'd need support from people with commit access, users and of course wip package maintainers.

Adam: Personally, I would like to have them regularly, maybe somewhere between two pkgsrc freeze, or something like that. It would be also good to have real life Hackathon, but that needs much more coordination and of course funds and time

pkgsrc is a complex system. How to get started as a pkgsrc maintainer, do you have any tips for beginners? 

Peter: Of course one should start by reading the Developers section in the pkgsrc Guide, but most important is to get hands-on experience. Just try and create a package for your favorite piece of software that is not in pkgsrc. Look at other packages and see how things are done there. If you can't figure it out on your own, IRC is a wonderful way to get support, from a great community.

One of the things newbies should remember that a lot of complexity in pkgsrc is in the software that is packaged. There is a lot of software with a lot of hardcoded assumptions that need to be patched and worked around. This provides for most of the complexity. Don't let this drag you down; try another package that is easier to package up, and when you have more confidence and experience try to tackle that difficult package again.

Adam: Read the guide and look at TODOs. Do small things like adding features to packages, correct bugs or upgrade package versions. Try to make a package and ask for reviews. It's best to learn by practice. Go to http://pkgsrc-wip.sourceforge.net for more information.

Are there any topics left you would like to talk about? 

Adam: One thing I would like to see is a separate bug-tracker for pkgsrc. Also, pkgsrc could be a very good testbed for pioneering new source control methods for NetBSD itself. But these are just my ideas, I don't know what is the official direction.


A talk with pkgsrc user Aleksey Cheusov

Hi Aleksey, please introduce yourself: 

My name is Aleksey Cheusov. I live in Minsk, capital of Belarus, this is an east Europe. I graduated from the Faculty of Applied Mathematics and Computer Science, Belarusian State University. And since then I work as a researcher/programmer. Most of these years I use Linux. And I'm not kernel hacker.

What is your motivation to use pkgsrc? 

It is very interesting to have only one packaging system for all operating systems that are in use. Contributions to cross-platform packaging system will always matter. This provides more freedom in selecting different operating systems: Which operating system you ever choose, you can always use pkgsrc. I'm confused why the MINIX 3 people still don't use pkgsrc :-)

Where and how do you use pkgsrc? 

At work I run shell servers running Linux. At home I use NetBSD as a workstation (video/music etc.) and for most of my development. Another big benefit of pkgsrc is its simplicity. IMHO it is much simpler for packagers than deb and rpm

Please tell us about the pkgsrc projects and the tools your currently working on: 

I've created the search tool pkg_online (wip/pkg_online). It uses a client/server approach. The main idea is to use a lightweight dictionary protocol (described in RFC-2229) for communication. Since 2002 I maintain the dictd project at http://sf.net/projects/dict and I think it is not only good for searching for natural words and their definitions/translations. pkg_online is a proof of concept. Both client and server are ready for use. A test server is up and running and listen dictd.xdsl.by:26280 But the information about packages is not updated regularly yet, a few preparations (cron jobs etc.) are still needed. Anyway you can try it.

There are many other tools I'm warking on. Use pkg_online_find 'm:s:cheusov' to find them ;-) But note that server is not up-to-date. Also see: These ones concern package management.

About package management: I think source-based upgrade is good for developers and have very important benefits. But for end-users and admins binary upgrades are preferable. It would be great to see a tool for binary upgrades as easy and powerful as Linux/Debian's apt is. I don't think it is very hard to create it. pkg_summary and pkg_src_summary is a main idea here.

I also would like to see support for Linux as good as it is for NetBSD and DragonFlyBSD. This may give life simplier for those who uses pkgsrc under this popular OS (like me). And this in turn may give more bug reports, more packagers/packages and more developers to this interesting project.

Are there things you like about other package building or package management systems? Any concepts or features you would like to see intergrated into pkgsrc? 

I'd like to see pkgsrc even more declarative. That is less statements like:
  post-install:
  ${INSTALL_XXX} YYY1 ${DESTDIR}ZZZ1
  ${INSTALL_XXX} YYY2 ${DESTDIR}ZZZ2
  ...

A better DESTDIR support (better packages coverage) is needed. Support for DESTDIR in bulk builds is also necessary.

You suggested to combine pkgsrc with the EPM packaging system What are the featuers and benefits of EPM? 

I didn't actually propose to integrate EPM to pkgsrc. At least because EPM is GPL-ed. This was just an idea. It can be interesting for vendors who provide solutions/software for different OSes. pkgsrc is ideal tool for building software on different OSes and EPM can be used to convert native pkgsrc packages to the native ones.


Towards an userfriendly Ports system: the MidnightBSD mports

MidnightBSD is a new BSD operating system, forked from FreeBSD, with focus on desktop functionality and ease of use (MidnightBSD project site). The mports system will combine the advantages of the FreeBSD and OpenBSD ports with features known from PC-BSD and other systems. A talk with Chris Reinhardt and Lucas Holt:

Hi Chris and Lucas, please introduce yourself: 

Chris: Well, I'm Chris Reinhardt. I'm the project manager for the mports infrastructure. I got into computer while I was at WPI working on a math major. Hard not to get into computers at WPI (Worcester Polytech). I got involved in a startup during the bubble. The company I was with was a FreeBSD shop. That came to end and I found myself in Ann Arbor, and I headed over to Eastern Michigan to finally finish my degree. Met Lucas. He needed *BSD hackers. I needed a project.

Lucas: My name is Lucas Holt. I am the founder of the MidnightBSD project. I am currently a computer science student at Eastern Michigan University. My first experience with BSD was at work 9 years ago. I started working with NetBSD on an old sparc in 2001 at home.

The MidnightBSD Project will have its own package build and management system, derived from the FreeBSD ports. Why didn't you choose one of the existing alternatives, pkgsrc, OpenBSD ports or the MirPorts? 

Lucas: When we started mports, we didn't see much activity with MirPorts. I didn't know much about the MirBSD project at the time. One of the other developers, Adam Karim, suggested OpenBSD ports. We agreed that we didn't want to include Perl in the base system and that our own system would provide us flexibility when developing an easy to use user interface for our target audience. I played with pkgsrc for a few days, but it didn't feel quite right. mports was initially meant to be a temporary situation until we could look at pkgsrc again. However, Chris came along and changed that.

What was wrong with pkgsrc? 

Chris: it's not that anything is wrong with pkgsrc, it's that we have some really specific goals with our ports system. pkgsrc doesn't target those goals.

Lucas: Initially, it was just quicker to hack up the FreeBSD ports system. I was most familiar with it. pkgsrc was architectured quite well, but initially getting it running was unpleasant. We just wanted a few ports to help out new developers. Adam was familiar with OpenBSD. Later, we decided that our own approach could target our goal as a desktop BSD much easier. In general, I think pkgsrc is quite nice. It seems to work well for NetBSD and DragonFly.

Chris:It became clear early on that the standard pkg_* tools weren't going to do what we want even if we integrated our tools with pkgsrc, we would be left with a hacked up version of pkgsrc that we would have to maintain.

Lucas: And our users may experience periods of instability that we hope to avoid.

What is the goal and the destination route of the mports system? 

Chris: mports can be seen as a cross between OpenBSD ports and FreeBSD ports. On OpenBSD using ports is discouraged, while package use is encouraged. On FreeBSD using ports to build from sources is the standard way a user adds third-party software, while the packages often have problems and are not used often by users. We don't want to go the whole way that OpenBSD does, but we want to encourage binary package use, our plan for that is to make really good package tools. Everyone knows the problems with pkg_add. It's slow because it does a two stage install with a forked off copy of tar. It's got a database that is nothing but a bunch of flat files. The code is hard to maintain.

Lucas: We would like to see the user experience on par with installing packages in PC-BSD, but without the wrapper setup used in PBIs (about PC-BSD and PBI). Improving the tools helps users and developers.

Chris: We're basically doing what anyone would do if they could rewrite pkg_add: we're using libarchive and SQLite. We're making a library that we'll link to graphical and command line interfaces. The library is really the big change. The existing graphical tools for package management are forking off executables. We can really integrate things for a rich user experience with libmport.

You mentioned the PBIs from PC-BSD. They require root permissions. Will it be the same with mports? 

Chris: Yes. In general, users have root access to their desktop machines :)

MidnightBSD aims to provide an easy to use desktop operating system. How will the new package build and management system meet this goal? 

Lucas: On most operating systems, users have difficulty installing software. The new system will make it much easier to install software on MidnightBSD. We hope that this will be similar to Linux distros like Ubuntu, but with the simplicity of installing software on a Mac or uninstalling software in WIndows. Our target audience will not be interested in using the ports system. Running make or even portupgrade in a command line environment is too confusing for many people. Experienced users running MidnightBSD will be able to use ports or packages depending on their preferences.

Chris: On the technical side building a library for package manipulation and improving the database makes it possible to build great interfaces to the package system. Lots of people use portupgrade just to get globbing. Using a SQL database gets us globbing for free.

Will mports integrate any concepts or solutions from other systems (pkgsrc, OpenBSD ports, ...)? 

Chris: We do a fake install much like OpenBSD. We haven't borrowed any ideas for pkgsrc yet, but that's only a matter of time. We're pretty unabashed when it comes to that sort of thing.

And are there any concepts from non-BSD or non-Linux systems that you would like to integrate into mports? 

Chris: Crazy as it sounds, but I see the GUI looking a bit like iTunes. Real time searching, with the zebra table of packages. We're also thinking about using the package system for operating system upgrades, much like OS X does.

Lucas: We've also talked about creating a software update dialog similar to OS X for system updates

MidnightBSD will provide a GNUstep-based desktop. GNUstep uses the concept of "Application Bundles" to store packages. How will this influence the filesystem structure and the package management of MidnightBSD? 

Chris: It really hasn't [had any influence]. libmport was really designed with about 4 goals in mind:
1. Speed

Everyone thinks about the speed of the actual install, but go to a FreeBSD machine and run make package in the xorg meta-port. Because of the way that the pkg tools work, the whole dependency tree is flattened into a list and passed to pkg_create. The system was never designed with a package with 200+ dependencies in mind.

2. Ease of new features

The pkg_* is a mess of spaghetti logic. Adding even small features is difficult. we got ride of all the flat file databases, and cleaned up the pkg file layout. Now we can do stuff like putting all the xorg packages into one package file.

3. Be a library (I've already discussed this)

4. Better remote fetching features

pkg_add -r works, but you can't browse, and it's all based on symlinks on a FTP server. We got some things coming down the pipe that will be much more powerful and easy to use.

Lucas: We can not use GNUstep applications for everything. The mports system has to be flexible with regard to software. Users want familiar applications like Firefox and Gimp as well.

Are there any topics left you would like to talk about? 

Chris: There's a number of smaller port system projects. MirPorts comes to mind. We're certainly one of the small ones right now. We're lucky though that thanks to Eastern Michigan University, we have a build cluster that we use to test the entire mports tree. Most of the smaller projects don't get to do that.

Lucas: Yes, the build cluster has been helpful in testing and debugging ports.

Lucas: Chris has done an excellent job on the software for the cluster. Thank you for this opportunity to talk about mports with you.


Simplified package management on GoboLinux

A tool for easy and comfortable package updates is one of the most wanted improvements for pkgsrc. The mostly named solutions are APT and YUM. But alternative implementations are available as well: The developers of Arch Linux preferred to create their own package manager PACMAN (information about Pacman). Even more interesting is the concept of GoboLinux: based on an alternative file system hierarchy a traditional package manager or related tools are no longer necessary (GoboLinux project page, another interview). A talk with GoboLinux developer Lucas Villa Real:

Hi Lucas, please introduce yourself: 

I'm in the development and users team since the early days, when Gobo was doing its first steps. I worked with Hisham and André in the same lab at the University, so it was more than natural to become involved with the project.

I started contributing with GoboHide, a feature which allows us to hide the legacy tree, kept for compatibility reasons. And with that I got the status of kernel maintainer for the distro, as I was the one who was in charge of keeping GoboHide always up to date with the latest kernel releases.

Also, I maintain a set of tools to create ports of GoboLinux for embedded devices, such as ARM, SH-4 and PPC-based ones. Nowadays I find myself working on Compile and Scripts very often, and also helping to review and commit recipes submitted by users in our mailing list.

Package management through the file system structure. Please tell us more about the concepts of package and software management on GoboLinux: 

Sure! The basic concept we have is that every package gets its own directory under a hierarchy named /Programs, and they'll have as many subdirectories as installed versions of the package, as in /Programs/GCC/4.1.1 and /Programs/GCC/4.1.2. All the files related to a given package should be within that single place.

Having this structure in mind, creating a package for an installed program is as simple as creating a tarball from its entry at /Programs, and removing it is achieved by removing its directory from there. The installation of a package is basically comprised by uncompressing it at /Programs and by subsequently activating it in the system.

After that we have the /System/Links tree, which holds subdirectories representing categories such as Executables, Libraries, Headers and Shared. Within each of these are symlinks pointing to every bin, sbin, lib, include and share contents from every active program at /Programs.

The activation step made after a package is uncompressed at /Programs just performs an update on those symlinks, making sure they exist and point to the new package's contents.

Besides the contents of the program itself, packages also contain information regarding run-time dependencies, required users and groups, environment variables that must be set, default configuration for settings file, etc. That information is processed by a set of scripts run at activation time, albeit it can still be done from the shell if one wants to play independent of them.

Which are the unique features and advantages of the GoboLinux package management system? 

With all this modularity, we don't need to keep in sync with a database to remind the user about which programs and versions are installed. All installed and compiled packages can be listed with ls /Programs, their versions with ls /Programs/Foo, and their contents with find /Programs/Foo/Version.

Dependency handling is also kept simple: a text file describes which programs and libraries a package depends on, and the check for these requirements is again made by looking at the /Programs hierarchy.

Being so clear and simple by design makes it a very powerful scheme to manage systems of different scales, as it's always possible to rollback the system to a previous state after a bad software upgrade and to have a consistent view of the system with no aid of special tools.

The /System/Links has an important rule, too. Since all legacy links point to their related categories inside that directory (ie: /bin, /sbin, /usr/bin and so on all point to /System/Links/Executables, /lib, /usr/lib point to /System/Links/Libraries), system indices can be centralized: $PATH is set to /System/Links/Executables, and ld.so.conf contains a single /System/Links/Libraries entry.

And with this, Gobo can be even more compatible than some more standard-looking distributions, as all standard paths work for all files, while other distros may struggle with incompatibilities such as scripts breaking with they refer to /usr/bin/foo when the file is actually in /usr/local/bin/foo.

Another property of the link-based indexing is that any references to non-existing files automatically become broken links, and therefore inactive. This makes it easy to spot and fix problems and, most importantly, ensures that the index is always in sync with the actual functional state of the system. No more problems of having the package manager complain that libXYZ is not installed even though you can see that it *is* there.

And what about the drawbacks? 

Today, the major limitation we have is that /System/Links can only point to a single version of a given package, as file names will conflict. Users have to either alternate between one version and the other using SymlinkProgram or to set environment variables such as PATH and LD_LIBRARY_PATH to do the same.

We are already working on a solution for this that will allow programs to have their own, private view of the file-system, reflecting the dependencies required by them. So, the view for a program that depends on LibXYZ 2.1 will be created by exclusively including that specific version in its view of the index tree (and that's only possible, again, thanks for the modularity of the /Programs dir).

Are there other package management tools necessary or at least useful on Gobo? 

Since packages are organized in a different way on Gobo, we have our own management tools for installing binary packages and compiling them from their source code. The most important ones are Scripts and Compile.

The Scripts tool takes care of all things system maintenance, such as installing and removing packages, signing and verifying their GPG signatures and parsing and taking action on the extra information carried by them (eg: checking dependencies and installing required users).

The Compile tool, as the name implies, is used to compile packages from their source code. It was designed in a way so that "recipes" to compile an application could be as simple as possible (recipes can be as short as 2 lines), increasing the availability of programs supported by the distribution through the voluntary contribution of users.

Both tools are encapsulated by a graphical front-end named Manager, which allows users to stay in the GUI while performing administrative tasks. Today more tools are being integrated with Manager's code base, such as Freshen, a tool that checks for updated packages and recipes and upgrades them.

Another project is the Compile build system: is it comparable to pkgsrc- or ports-like systems? What are the differences and advantages? 

Yes, it is. All these port-like systems are doing in effect the same job: they read a description of how to download, compile and prepare the system to run a package. Every implementation defines its own pseudo-language and calls methods and scripts that potentially will only exist in that very distribution, and Compile is no different in that sense -- it just applies to the way we organize programs in GoboLinux.

What differs Compile from other build tools is that we chose to move all the intelligence on how to compile a program to inside the tool,making the compilation recipes as simple as possible. The user needs to define a download location (url, svn, cvs or git) and the compilation method, which will trigger a method depending on it: "makefile", "configure", "perl", "xmkmf", "scons" and "python" are some of them that will invoke the respective Compile functions to compile and install the program.

In general these 2 lines are sufficient for many programs, but Compile can also invoke user-defined functions and parse extra variables that might be required to install more tricky ones.

Is the Gobo package management and build system for finished? If not, which improvements and changes are projected (or already under development)? 

We have just started to work in an automated build system for generating the whole distro in batch. It should be able to create packages for different profiles (different architectures and base system) for any recipe in the GoboLinux tree.

Also, package management is under constant change: FreeDesktop.org is still defining standards for better desktop integration, and we need to adapt our tools for that. And with the incoming of the new file-system for the private view of the /System index we'll have a lot of work to do, too.

I don't know if we'll ever see the day where we would consider the package management "finished", as the elements involved in the compilation of packages are under constant change. At the best we can tag it as "stable".

Are you confronted with objections to your package management and building concepts? 

No, nothing that can be considered seriously. Most people objecting to Gobo's concepts are not really aware of what the proposals are, or just don't like the idea of changing the way package is managed because the one they're using works fine for them. The complains hardly go outside this scope.

What do you like about other package building and management systems? Any concepts you would like to see ported to Gobo? 

Right now we're in the process of designing "USE flags" for Compile, as seen in Gentoo's Portage. That will give the build system the ability to switch over one or another feature that the user might want to give preference to -- with no intervention from the user in the compilation process.

As commented earlier, we're also working in an implementation to give packages a private view from the filesystem. That's a similar concept to Plan 9's local namespaces, except that we're doing that in userspace with FUSE.

For the time being these are the most important changes that we'll have in the near future.

What do you think about package management systems like "Zero Install"? Would it be a useful and feasible addition to Gobo? And how could it be integrated? 

There's a couple of GoboLinux users which already happens to run Zero Install on it. However, that's a complement which needs to be managed by Zero Install itself -- GoboLinux tools aren't able to help on its maintenance.

We provide an environment named Rootless for those who want to compile and install packages in their home directories. Relocatability is a problem for some binary packages, though, as references to file names and directories usually end up hardcoded into the executables. But the infrastructure works like a charm for source code based installs, and allows end users to use our tools in any environment where a shell is available, even in other distributions or operating systems.

Are there any topics left you would like to talk about? 

We'd like to thank you for the opportunity of this interview, and invite anyone curious about our different approach towards Linux systems to check it out by giving our Live-CD a spin. The newly released version, 014, is available for download at http://www.gobolinux.org


The Zero Install system

Thomas Leonard is software developer and well known as the creator of the ROX desktop (running on X11). He also invented the Zero Install system (Zero Install project page), aimed as an easy way to download and upgrade applications. Zero Install is based on the Python programming language and operating system independent. (Articles: 1, 2):

Hi Thomas, please introduce yourself: 

I'm been programming since I was 8, starting on a BBC Micro. I got involved with OSS when I ported Vim to RISC OS. I got my PhD in computer science at the University of Southampton, where I currently work. While studying, I created the ROX desktop, which aims to bring some of RISC OS's user interface features to Unix and Linux systems. My most widely used software is probably the freedesktop.org shared MIME database, which contains the rules for working out a file's type.

Since when do you develop the Zero Install system and what was your motivation to start the project? 

I started it in 2003. The ROX desktop is made up of a number of software packages by various people, each of which is a RISC OS style 'application directory'. Like bundles in MacOS X, each package becomes a directory in the filesystem, which is run by clicking on it.

Application directories have some nice properties: they don't conflict with each other; they don't ask you to run anything as root to install them; they never refuse to uninstall; anyone can distribute them; and anyone can install them.

The problem was that downloading and unpacking all these programs was a fair bit of effort, and keeping them up-to-date was worse. Also, if you had a multi-user system then each user had to do this individually (or they had to trust each other).

Zero Install is a way to preserve those nice properties, whilst still getting the now-expected features of a proper installer, such as automatic upgrades, dependency handling, and digital signatures.

What are the features of Zero Install? 

Zero Install is a decentralised installation system, which means that there is no central repository. Every package is identified by full URI rather than by a short name. You can create a new library and put it on your web-site, and then someone else can write a program that depends on it. You don't need to ask me to add you to any kind of central index of software before people can start running the program.

It resolves depencies on a per-program basis, not globally for the whole system. If you run two programs that need different versions of ROX-Lib, for example, it will just download two copies. But, it's efficient: if both programs should use the same version of ROX-Lib, it shares that one version automatically. You need this in a decentralised system; you can't have one unmaintained package preventing others from being upgraded, and you can't have the old package stop working, either.

All package descriptions are GPG-signed, and contain cryptographic hashes of the contents of each version. Each version of each program is stored in its own sub-directory and no code from the package is run during install or uninstall. It allows unprivileged users to install and share software automatically, even if they don't trust each other.

I think Zero Install's approach to security is fairly unique (Java Web Start is similar, though). It tries hard to ensure that installation doesn't get the chance to do any damage. If you create a malicious package, I should be able to install it on my machine, so that the program appears on my Applications menu, without your package being able to do any damage. The long-term goal is to combine this with run-time sandboxing, but Zero Install only covers the installation part.

One feature of Zero Install is the platform-independency: On which operating systems does it work? 

Any Linux or Unix-type system. There are packages for Fedora, SUSE, OpenSUSE, Debian, Ubuntu, Slackware, FreeBSD, Gentoo, Solaris and MacOS X. It could probably be made to work on Windows too, but we can't seem to find any Windows programmers.

Tuomo Valkonen, software developer and a critc of package systems, criticizes "certain WIMP-centricism", a "primitive" dependency handling and the missing search mechanism for packages (link): 

Zero Install has a console and a GUI interface, but the GUI has far more features, reflecting its origin as a distribution mechanism for a desktop environment.

Features get added as they become necessary. When there are enough packages that you need a search engine (we have about 150 currently), I'm sure one will appear (freshmeat maybe?). I think it's more likely that you just search for software first and then figure out how to install it, rather than trying to figure out what you can install with a particular tool.

I find Tuomo's window manager, Ion, very interesting. In fact, I've been using it for a few months now. A major challenge for designing a secure desktop environment is to allow running GUI programs without giving them unrestricted access to my account. If applications are free to open windows whereever the want, you don't really have much control. Tiling window managers are a possible solution to that.

He also suggests some changes: replacing Autoconf (and related stuff), integration of a sane build tool, a "capabilities" system and more (links: 1 , 2): 

When you ask Zero Install to compile from source, it fetches the source code, and any dependencies (build tools, header files, etc) and then runs a build script provided by the source package, so you can use whatever build system you like. The autotools are pretty bad, indeed, but there's a tutorial here showing how to use it with SCons, for example: http://0install.net/0compile-scons.html

Historically, people haven't embraced new build tools easily because it puts users off trying your program if they need extra bits to compile it. Being decentralised, you can use Zero Install to depend on something like SCons (http://www.scons.org) and have it available to all your users, whatever their distribution.

pkgsrc is also cross-platform, could it be an appropriate build system for Zero Install? 

We rely on people with experience of different systems to let us know if there's some cool feature we're missing. At the package level, compilation seems pretty simple, though: you run a script which takes a read-only "source" directory and populates an empty "target" directory Then you tar up the target directory and you've got your binary package!

You follow the development of other new package management and decentralised installation systems very observingly: Which other systems do you like (and why)? Are there any features which you intend to adopt for Zero Install? 

Well, Nix (project page) has garbage collection already, which we'll need for Zero Install one day (you have to uninstall programs one-by-one currently). It's low priority at the moment because available disk space is typically much larger than then amount of software available through Zero Install! I hope the Autopackage guys keep pestering the Python developers and others to take binary compatibility seriously too.

I'm also interested to see what the Klik developers come up with for Klik2 (project page). They're using the Zero Install XML feed format now and they're doing something clever with union filesystems to enable non-relocatable binaries to run unmodified.

What do you plan for the future development of Zero Install? 

Automatic support for mirrors. Peer-to-peer downloads, binary deltas, more sandboxing. See the Roadmap.

Are there any topics left you would like to talk about? 

I'd to encourage people to have a play. People seem to worry about messing up their system with two package managers, but it doesn't interfere with anything else and you can always create a new user and install it to $HOME if you're not convinced: http://0install.net/injector.html