While working on the xapi rebase on top of xenserver’s updates, I’ve observed a vast amount difference in time spent over the packaging between xenserver and xcp-ng, I believe the process used right now drains a lot of engineering time, and we should rethink how we approach our relationship with Xenserver with regards to packaging now that we are becoming well-resourced to be able to get much more value from the packaging process.

I want to present an analysis of my packaging experience of the toolstack rebase and hope we can spend time discussing the ideas here to change the current approach to packaging.

Here are the analysis and ideas to move forward that I wrote, stemming from the process:

Expectations from before doing the work

• on XS, xapi packaging is mostly mechanical, it’s quite automated and it only need manual attention once every few months, this should be similar. For reference, the whole process of branding a release and run the baseline of end-to-end testing takes less than 6 hours.
• The value of packaging is delivering features and bugfixes to customers, and letting customers know that both are present in the release. This is done by integrating unit-testing, unlocking end-to-end testing, and linking the contents of the packages to the features and bugfixes using a changelog, using ticket names (CA-/CP-). This is made evident by knowing the size of the team maintaining packaging metadata in XS: 0
• Following xenserver’s packaging won’t take most of my time. This is because in XS not much attention is given when code is being packaged, value is considered to be in other activities. This means spending time in XS’s packaging is time that could be better spent doing other activities that bring more value to xcp-ng. (fixing issues, working on customer issues, pushing features)

What happened

• The process to integrate new package versions from xenserver’s toolstack (ocaml-based packages) has taken more than a week.
• This week has been spent by
• Updated one package ahead of XS, this brought changes on a dependency (~1 day)
  *handling complexity in the process of preparing PRs: Preparing the base branch, which way to do a merge, change the versioning metadata, handling divergence from XS upstream packages. (~4days)
• New packages were released from Xenserver while I was working on the previous release. Xenserver can have a weekly release cadence.

The value of Xenserver as upstream

Currently, great care is put into following xenserver as upstream, but little care is put within xenserver at the time of packaging, in most cases. This is because of the perceived little value spending time on it provides. There’s one big exception, where great packaging effort is spent within xenserver: xen. This is because of the great divergence between upstream (xen project) and xenserver. This means that there’s value in xenserver’s packaging of xen, and makes sense to use it as upstream.

But this is not the case for other packages with a different upstream, particularly with xapi-project. Instead of following XS’s packaging, we can package the releases coming directly from xapi-project repositories, this has several benefits: - The team can spend time making packaging decisions that best fit XCP-ng, instead of compromising with whatever decisions that best fit xenserver. - Packaging early means we can have 3-4 weeks to test and provide fixes to upstream before xenserver releases packages, and can end up in a simultaneous release with them, instead of releasing weeks or months after them. - We can work on reducing the amount of patches between upstream and xcp-ng packages directly at the upstream level, instead of at the packaging level. Non-issues: - Security patches: we’re part of the security effort of xen and xapi projects, so we can preempt the patches and deliver them without depending on xenserver’s srpms. - Long Term Support: xenserver engineers develop on xapi-project repositories, xcp-ng can do the same, we have the power to add patches to LTS branches as well. Possible risks with the approach: - Xenserver keeps some patches on xapi to adapt to different xen versions, including their custom ones. There’s effort to have conditional compilation in xapi to remove the dependency on the xen patches and make it work in all cases, but this is untested.

Other ideas

There could be automation that uses the metadata in the XCPNG tickets that is used to create tickets for XO, generate forum posts to let users test the changes, etc