Server Working Group Weekly Meeting (2014-09-09)

#fedora-meeting-1: Server Working Group Weekly Meeting (2014-09-09)

Meeting started by sgallagh at 15:00:56 UTC (full logs).

Meeting summary

  1. roll call (sgallagh, 15:00:56)
  2. Agenda (sgallagh, 15:04:17)
    1. Agenda Topic: ARMv7 Support (sgallagh, 15:04:17)
    2. Agenda Topic: Fedora 21 Alpha Announcement and Download Page (sgallagh, 15:04:17)
    3. twoerner wishes to discuss rolekit remaining tasks, but we probably don’t have enough time with the relevant people at this meeting. (sgallagh, 15:07:16)
    4. Agenda Topic: Netinst media update (sgallagh, 15:07:51)

  3. ARMv7 Support (sgallagh, 15:08:09)
    1. (sgallagh, 15:09:03)
    2. We need to be able to test ARMv7 hardware to make certain it’s not completely broken (sgallagh, 15:11:59)
    3. nirik has some available hardware (sgallagh, 15:12:09)
    4. pwhalen reports that ARM is quite usable (sgallagh, 15:12:32)
    5. PXE-based installs are working for general F-21 on ARM, so Fedora Server should be in reasonable shape (sgallagh, 15:13:10)
    6. pwhalen has been testing network installs on a couple of hw targets, wandboard, cubietruck and highbank (sgallagh, 15:14:38)
    7. pwhalen confirms that he’s been testing each TC on ARM. We’re in good shape. (sgallagh, 15:15:15)
    8. ACTION: pbrobinson will also help test various devices on Fedora Server (sgallagh, 15:15:32)
    9. No known ARM-specific blocking issues at this time (sgallagh, 15:16:32)

  4. Fedora 21 Alpha Announcement and Download Page (sgallagh, 15:17:45)
    1. (sgallagh, 15:17:45)
    2. (sgallagh, 15:17:45)
    3. We want to add mention of both Cockpit and rolekit as foundational tools of the Fedora Server on the Alpha announcement (sgallagh, 15:25:13)
    4. ACTION: simo and stefw to take a first crack at the text and send to the mailing list for tweaking. (sgallagh, 15:25:43)
    5. AGREED: Content of the Fedora Server download page looks good and is approved by the Server WG (sgallagh, 15:29:26)

  5. Netinst media update (sgallagh, 15:29:52)
    1. see for details on what’s needed to make per-Product network install images work. our involvement is likely to be limited to monitoring and testing (adamw, 15:43:27)
    2. adamw will communicate once we have a TC/RC where the new netinst bits should be in place and folks can test/examine the mechanism (adamw, 15:54:49)

  6. Open floor (adamw, 15:55:46)
    1. we note the sixteen-wheel truck on the horizon labelled Lennart’s Latest Bright Idea – (adamw, 16:04:28)

Meeting ended at 16:15:29 UTC (full logs).

Action items

  1. pbrobinson will also help test various devices on Fedora Server
  2. simo and stefw to take a first crack at the text and send to the mailing list for tweaking.

Action items, by person

  1. pbrobinson
    1. pbrobinson will also help test various devices on Fedora Server
  2. simo
    1. simo and stefw to take a first crack at the text and send to the mailing list for tweaking.
  3. stefw
    1. simo and stefw to take a first crack at the text and send to the mailing list for tweaking.

People present (lines said)

  1. sgallagh (71)
  2. simo (59)
  3. adamw (58)
  4. nirik (12)
  5. mitr (8)
  6. mizmo_ (8)
  7. pbrobinson (6)
  8. pwhalen (5)
  9. zodbot (5)
  10. twoerner (5)
  11. stefw (4)
  12. tuanta (3)
  13. davidstrauss (0)
  14. mizmo (0)

Generated by MeetBot 0.1.4.


Server Working Group Weekly Meeting (2014-08-19)

Meeting started by sgallagh at 15:00:17 UTC
(full logs).

Meeting summary

  1. roll call (sgallagh, 15:00:17)
  2. Agenda (sgallagh, 15:03:35)
    1. Agenda Item: Status of rolekit (sgallagh,
    2. Agenda Item: Status of Cockpit (sgallagh,
    3. Agenda Item: Open Floor (sgallagh,


  3. Status of rolekit (sgallagh, 15:04:23)
    1. Major kudos to twoerner and mitr here
    2. rolekit in decent shape for F21 Alpha. Clean-up
      and stabilization work will happen for Beta.
    3. ACTION: nirik, tuanta
      and handsome_pirate to look into the Database Server Role


  4. Status of cockpit (sgallagh, 15:14:39)
    1. Cockpit upstream expects to be stable in time
      for Fedora 21 Beta. A testable version is available today for Fedora
      21 Alpha


  5. Open Floor (sgallagh, 15:24:52)
    1. We expect a Fedora 21 Alpha TC3 build
    2. ACTION: mitr and/or
      sgallagh to test the server presets on Alpha TC3

Meeting ended at 15:36:51 UTC
(full logs).

Action items

  1. nirik, tuanta and handsome_pirate to look into the Database Server Role implementation
  2. mitr and/or sgallagh to test the server presets on Alpha TC3

Action items, by person

  1. mitr
    1. mitr and/or sgallagh to test the server presets on Alpha TC3
  2. nirik
    1. nirik, tuanta and handsome_pirate to look into the Database Server Role implementation
  3. sgallagh
    1. mitr and/or sgallagh to test the server presets on Alpha TC3
  4. tuanta
    1. nirik, tuanta and handsome_pirate to look into the Database Server Role implementation

People present (lines said)

  1. sgallagh (58)
  2. nirik (17)
  3. danofsatx (10)
  4. puiterwijk (9)
  5. tuanta (7)
  6. mitr (5)
  7. zodbot (5)
  8. simo (3)
  9. twoerner (2)
  10. robyduck (1)
  11. mizmo (1)
  12. adamw (0)
  13. stefw (0)
  14. davidstrauss (0)

Generated by MeetBot 0.1.4.

Fedora Server News Update

Well, we’ve certainly been neglecting this blog lately, haven’t we?

A lot has been happening these last several months on Fedora Server. As we approach the Fedora 21 Alpha date (delayed several times due to events outside our control), now is a good time to look back at the progress that we have made.

I (Stephen Gallagher) gave a detailed talk at the recent Flock conference in Prague on the state of the Fedora Server, highlighting the three primary technologies that we are working on. My friend and colleague Paul Frields wrote up an excellent summary (with link to the video) over at the Fedora Magazine. Check it out!

Fedora Server Role D-BUS API Design Discussion

During this week’s Fedora Server Working Group meeting, we started planning out how we want the public D-BUS API to look. I volunteered to take the discussion to the list so we could start hashing it out with a wider audience.  I’d prefer to keep replies on please (easier to track on one list), but if comments are made on this blog post, I’ll forward them along as well.

Some pre-conditions/goals (many of these are reminders from earlier discussions):

  1. 1) We want to be able to support two initial clients of this API: the Cockpit management console and a local command-line tool (I’ve been tentatively calling it ‘fedora-role-manage’) A third, future option will be to also support remote access to this API via the OpenLMI Project.
  2.  We do not need to have a single generic API that will work for any conceivable Role. We’ve identified in the past that the selection of Featured roles will (for the conceivable future) remain a small, manageable list. For this reason, it is acceptable for the deployment and post-deployment configuration APIs to be entirely different between two roles. However, some functionality such as the backup and firewall queries should be stable between all roles.
  3. The supported clients should be able to determine if a target system supports the deployment of a particular role (for backwards-compatibility and architecture differences), but the Role API should *NOT* be attempting to dictate the UI experience. UIs should merely check whether the role is available and then use a previously-developed experience. We don’t want to be reimplementing PAM or debconf here.
  4. The D-BUS API must be stable. If it is likely that the set of input required to deploy or configure will change, this should be planned for in the design.
  5. As it is not possible to plan for all contingencies, the API must be versioned such that the clients can determine whether they can properly perform the requested actions.

Some initial thoughts on the matter (all are open to discussion):

Standard Objects:


  • properties:
    • all_roles: list of object references to Roles that can be installed and deployed
    • staging_roles: list of object references to Roles that have been pre-loaded but not yet configured
    • active_roles: list of object references to deployed Roles
    • version: API version
  • methods:

These properties and methods should be available and identical on all Roles


  • properties:
    • version: API version
    • loaded: Bool: whether the packages are installed
    • deployed: Bool: whether the Role has been deployed
  • methods:
    • PreloadRole: Install all necessary software packages for default operation
    • GetFirewallPorts: list of port/protocol pairs that the Role needs open in the firewall
    • PrepBackup: method to invoke any necessary pre-backup steps (such as running a database dump to a file)
    • GetBackupFiles: list of filesystem path objects that identifies all data that should be copied by backup software

Individual Roles should have unique implementations of installation and deployment

e.g. /org/fedoraproject/server/role/DomainController:

  • methods:
    • DeployPrimaryDomainController: Set up the first domain controller in the domain.
    • DeployReplicaDomainController: Add a new replica domain controller to the domain
    • AddCertificateAuthority: Add a certificate authority to this domain controller
    • EnableDisableDNS: Enable or disable the DNS server on this server

Open questions

  1. How do we handle configuring the Firewall? I think we want to have a Firewall object (/org/fedoraproject/server/RoleFirewallManager) available to query the firewall as a whole, but we may also want to be able to view and apply firewall rules from the Role objects directly. (Note: I’m not suggesting we need a complete firewall solution here. This should be a wrapper that deals only with the Roles). Firewalld already provides a more comprehensive firewall interface for the general case.

I’m sure there are other questions we’ll discover out there, but this email is already running long for an “introduction” to the topic.

Fedora Server Technical Specification Working Session (Day One)

Day One

Yesterday, the Fedora Server Working Group met to put together the Technical Specification, as required by FESCo and the Fedora Advisory Board as part of the initiative.

Supported Architectures

After some discussion, the working group agreed that Fedora Server will initially support the ix86, x86_64 and armv7hl architectures. In the future, once hardware becomes readily available, Fedora Server will support 64-bit ARM achitectures as well.

Partitioning and LVM

An hour-long discussion was held on the selection of the default, guided storage path. We determined that there were several points that need to be discussed with the Anaconda and user-experience teams:

  • How difficult is it to have separate guided paths for the separate projects?
  • Is it acceptable to reduce the guided path to a single choice for the Server Product?

It was agreed by the Server WG that we would prefer the option for a single guided storage path. We also agreed that this preferred option would be to use the XFS filesystem on LVM.

The reasoning for the XFS choice is that we are reusing research performed by Red Hat that showed that XFS was the best-performing filesystem for the majority of server loads. Additionally, we want to support LVM for a variety of reasons, including but not limited to easy expansion of logical volume size and the potential for snapshotting.

We discussed the possibility of using LVM thin-provisioning instead of classic LVM, but reached no consensus. Discussion of this point will continue as we go forward. For Fedora 21, it will be mostly an implementation detail.


The selection of the default firewall-manipulation solution was fairly straightforward. A few questions were raised on the list about the reasoning for using a daemon instead of the traditional iptables rules. However, the advantages to firewalld are several:

  • It provides a relatively simple public API that can be consumed by projects such as OpenLMI to be able to manage the firewall rules separately.
  • The command-line tools for firewalld are much simpler to use for the common cases (simple port opening/closing).
  • It is expected that the firewalld interfaces will remain constant when the kernel changes from iptables to nftables.

Server Desktop

Much discussion was had over how to handle a graphical console for the Fedora Server. Some of the WG counseled deferring an official answer on this until Fedora 22. Others argue that we need to have at least some answer to this, since there exist third-party applications that cannot be installed without a console.

No consensus was reached on a selection of desktop environment. We instead decided that we would simply provide comps groups for each of the desktop environments currently available in Fedora and provide no direct guidance or support for any of them at this time.

Software Installation

Related to the previous discussion about desktops, we also agreed that we will not restrict the install media (particularly the network install or pointing Anaconda at additional repos) from being able to install any packages from the wider repositories. Fedora Server SIG does not commit to supporting anything outside of the standard install and Roles beyond the support already provided by the Fedora community.

Day Two

I put together a rough draft of the Technical Specification on the wiki. There are a large number of open questions, so we will be having a second meeting of the working group today (2014-02-28) at 1500 UTC to go through them.

Primary Goals for Fedora Server in Fedora 21

At today’s Server Working Group meeting, we affirmed the vote that’s been happening on the mailing list regarding the first two Server Roles that we plan to package for the Fedora Server.

Domain Controller

We unanimously selected a “Domain Controller” Role (powered by FreeIPA) as our primary, blocking deliverable for the first release of the Fedora Server. There are several reasons behind this decision. First of all, the FreeIPA project already possesses many of the qualities that we desire in a Server Role: it glues together all of the technologies that it needs to make work, it can be installed very easily and it serves a clear, high-level purpose.

In addition, the Domain Controller Role will serve as a fundamental piece of the architecture for interacting with the rest of the Fedora Server Roles and ecosystem. By setting up and enrolling servers in a domain, we will be able to take advantage of single-sign-on capabilities and other centrally-managed identity features in our other roles, including the ability to manage the set of administrators allowed to deploy roles on machines.

Database Server

We also unanimously agreed that the second Role we would develop would be a Database Server. This was chosen because, like the Domain Controller Role, most further roles will rely on the presence of a stable, usable database. The selection of *which* of the open-source databases to use was slightly more diverse, with seven members of the Working Group favoring PostgreSQL, one preferring MariaDB and one abstention. As a result, we will proceed with PostgreSQL as the initial provider of the Database Server Role, with an option in the future to develop and support other databases as well.

The Database Server is being classified as “highly desirable”, but it is not a strict blocker to the first release of Fedora Server (the way that the Domain Controller Role is). We will try as hard as possible to ship both Roles in the first release, but if we one has to be deferred, it will be the Database Server Role.

Fedora Server PRD Draft – Final Call for Feedback!

As Stephen pointed out last week, we’re getting really close to the Product Requirements Document (PRD) deadline for In our meeting yesterday, we cleaned up our to-dos and polished things off in the Fedora Server PRD.

You can read our draft below. Please, comment here with any feedback / suggestions you may have, or as always participate on the Fedora Server mailing list or the #fedora-server IRC channel on


Fedora Server PRD Draft

Document Purpose and Overview

What this document describes

This is the Product Requirements Document for the Fedora Server. It:

  • Provides a high-level market overview of the infrastructure server market as it pertains to the Fedora Server; this includes items which may not be within our actual scope/ability to accomplish at the current time.
  • Provides deeper understanding of the types of users who could use Fedora for their application and infrastructure needs. This includes describing their main day-to-day tasks, common problems, etc. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
  • Ties common issues and needs of potential users/consumers of the Fedora Server product to high-level product needs, from a “functional” standpoint.

This document does not dictate implementation details. The goals in this document will drive the continued implementation of this product.

Fedora Server Vision Statement

Fedora Server is the preferred [community] platform for system administrators and developers seeking to deploy applications and services that use the latest technology on a stable foundation with effective resource utilization.

Fedora Server Mission Statement

Fedora Server is a common base platform with “featured server roles” built on top of it. We commit to produce, test, and distribute these server roles.

Market Opportunity

The server operating system market is mature, but constantly evolving. Fedora Server has opportunity for adoption as people deploy new applications and services as well as migrate legacy applications to new platforms.

With the increased complexity of IT and software, system administrators find it more difficult to devote large amounts of time to dealing with individual technologies, specifically:

  • Detailed understanding of the underlying technology, as has traditionally been required for Open Source server solutions, is often not feasible.
  • The constant focus of Fedora to ship the latest upstream releases of software, combined with a short lifecycle and lack of tooling, require frequently spending significant amounts of time migrating from one Fedora release to another.

These impositions on system administrators are a significant barrier to further growth of Fedora, and we believe that by improving the product we can remove these requirements and barriers, opening Fedora up to a much larger user base that has so far been limited to non-Linux server solutions.

Fedora, being a leading-edge Linux distribution, is the ideal place to design such server improvements for the Linux ecosystem together with real-world users to make them available to early adopters.

Product Objectives

Fedora Server consists of a release that can be used at various scales. For example, it could be used in the following example environments:

  • A single server that runs an application or service.
  • A platform for an entire data center.
  • The platform to deploy an Infrastructure-as-a-Service (IaaS) private cloud.

Fedora Server is meant to be used for the following purposes:

  • An operating platform for important infrastructure tasks (DNS, DHCP, etc.) and server applications (file/storage server, database server, user-developed web applications, etc.)
  • A platform for deploying Infrastructure-as-a-Service systems for best deployment of Fedora Cloud images.
  • Infrastructure to allow efficiently managing many servers as a single unit. The product only commits to producing basic tools, but the infrastructure will allow more advanced tools to be created.

The Fedora Server product will also provide one or more “roles” that can be used to easily spin up a service or application. Roles will be introduced gradually as development and testing allow, but it is a primary objective to offer useful roles built on top of the base server offering. More information about what roles are is provided elsewhere in this document.

Secondary Objectives

Aside from the adoption and development of applications on top of the Fedora Server platform, we have a few secondary goals that should be helped by wider adoption:

  • More testing of Fedora images with additional bug reports.
  • Better feedback about product direction and potential improvements. This is separate from “bug reports” in that we hope to engage the audience and receive detailed feedback about use cases, desired features, developing trends in cloud management, etc.
  • More patches and contributions that will help improve the product, and Fedora in general. Assuming we’re successful, some users should take an interest in helping to develop our product.

User Profiles, Primary Use Cases and Goals


We will use a set of personas to describe our target users and their respective needs. This document will list the personas in their simplified forms, with detailed explanations of each one available on their respective wiki pages.

1. System Administrator “Macgyver”
Administrator with limited hardware and personnel resources to work with
Needs to be able to do “a lot with a little”
2. DevOps Engineer/Administrator
Focus is on time-to-deploy and time-to-recover as opposed to uptime
Value is achieved by delivering the latest capabilities fastest
Needs to be able to deliver quickly to PaaS, SaaS and bare-metal servers
3. Traditional Application Developer
Needs a platform with API and ABI stability guarantees
Focus will be on minimizing risk when making changes
Cannot tolerate rapid changes in core functionality
4. Junior Enterprise System Administrator
Simplify automation to manage repetitive tasks
Needs intuitive mechanisms to effect common changes
5. Decision Maker
Makes purchasing decisions and directs technology choices
Interacts with upstream FOSS communities to identify potential value

(The full Personas document is available on the Fedora Project wiki.)

Use Cases

The Fedora Server will need to address the following use-cases:

  1. The user must be able to easily deploy and configure any supported Fedora Server role. (Examples may include: FreeIPA Domain Controller, BIND DNS, DHCP, Database server, iSCSI target, File/Storage server, OpenStack Hypervisor Node.)
  2. The user must be able to query, monitor, configure and manage a Fedora Server remotely using stable and consistent public interfaces.
  3. The user must be able to simply enroll the Fedora Server into a FreeIPA or Active Directory domain.
  4. Users must be able to control and contain the resources consumed by services running on the system.
  5. Users must be able to rapidly re-deploy services in accordance with their DevOps practices using Fedora Server.
  6. Users must be able to create, manipulate and terminate large numbers of containers using a stable and consistent interface.
  7. The user must be able to install and manage Fedora Server in a headless mode where the display framework is inactive.

Featured Server Roles

A Featured Server role is installable component of Fedora Server that provides a well-integrated service on top of the Fedora Server platform. These prepared roles simplify deployment and management of a service compared to setting up an upstream server from scratch; their use is recommended but optional; existing users of upstream servers based on Fedora RPMs will not be impeded.

Initially, we will test and support only a single role per server; installing several non-conflicting roles is expected, but we will not be testing every combination.

Mandatory Requirements

Mandatory requirements are exactly as they sound: if any one of these requirements is not met, the server role is ineligible for elevation to “featured” status.

  1. The Server Role must be packaged in such a way that it is possible to install the complete set as a unit. AKA “One-click Install”. [Use Case 1]
  2. The Server Role must provide an API or similar stable external interface for configuring the role post-installation. Exceptions to this rule may be granted if and only if Optional Requirement 2 (below) is met and it can be demonstrated that the default method of operation is the only reasonable method of operation. [Use Case 1]
    1. If the API is provided by the Fedora project it must be built using a common framework to all Fedora developed Role APIs. If upstream provides such an API, it is permissible to use that.
  3. The Server Role must provide an API to interrogate the configurable settings of the role as identified by Mandatory Requirement 2. [Use Case 2]
  4. The Server Role must add to the functionality of the Fedora Server. In other words, a Server Role cannot be considered “Featured” if its purpose is to further reduce the minimal system.
  5. A Server Role must have an identified point of contact that agrees to maintain the Server Role in Fedora. If this person or group becomes unresponsive, the Server Role will necessarily be demoted from Featured status. A Server role must be possible to trivially update to fix security vulnerabilities, and maintained to provide security fixes timely.
  6. A Server Role must be installable without a local graphical utility. [Use Case 7]
  7. If a Server Role packages multiple upstream projects together as a suite or solution, updates to any of these projects must be coordinated for concurrent release to allow testing of the complete system.
  8. A Server Role must automatically use and sufficiently implement $specified system-wide settings (e.g. a LDAP source for account information, CA certificates, and similar domain-originated or system-wide configuration)

Conditional Requirements

This is not a list of things you should ignore. Rather, this is a list of things that if you implement them, you are committing that they must continue to behave that way in the future.

  1. The Server Role may be packaged in such a way that it can be uninstalled as a complete unit. The mechanism to accomplish this removal must remain consistent. [Use Case 1]
  2. The Server Role may provide a sane, usable default. If it does so, it must remain capable of accepting input through its configuration API to change this default. [Use Case 1]
  3. The Server Role may provide an API to interrogate additional useful information about the running service(s) provided by the Server Role. The content here is up to the Server Role to define, but should be treated as stable (e.g an update should never remove the ability to monitor something). [Use Case 2]
  4. The Server Role may provide a specification of its needed resources to the Fedora Server in a standard format. (e.g. Memory requirements, etc.) If this is provided, the Fedora Server may use this information to create isolated services such as VMs or containers in which to run the Server Role. [Use Case 4]
  5. The Server Role may request operation in an isolated environment (with or without implementing Optional Requirement 4). If it does so, the Server Role must indicate (using a standard Fedora Server API) what form(s) of isolation it supports. Fedora Server must honor this. [Use Case 4]
  6. A Server Role may provide a remote graphical (which includes web-based) configuration tool.
  7. A server role may implement all of the other requirements by integrating with the Fedora Server-designated infrastructure (commands, APIs, and the like). This is preferred but not mandatory if the upstream provides a different way to achieve the same objective.

Logistical Concerns

Delivery Mechanisms

Fedora Server will produce 2 main installation resources, a netinstall image and an offline install iso image that allows to install and configure featured roles offline.

Supported installation methods:

  • Automated (“mass”) install within a larger Linux infrastructure (e.g. PXE is an option, may have LDAP/IPA).
  • Manual install without a supporting infrastructure (e.g. the very first Linux server).
  • Where possible, existing servers and installed roles should be upgradable to new releases with minimal involvement by the admin.
  • Individual roles should be installable at server install time (e.g. in Anaconda) and post-install.
  • Where to obtain

    Users will be able to obtain these images from the Fedora Project website and mirror networks.

    Measuring Success

    In order to measure success we will monitor (somewhat arbitrary) numbers over time. The list of metrics we take in account will be adapted over time to measure specific efforts within the framework of the Server Working Group goals.

    The initial basic set of metrics will be:

    • Usage of Fedora Server images in AWS or other public cloud providers that publish such aggregated data.
    • Number of contributors that explicitly contribute to the Fedora Server effort in some way. We’ll try to gather data from sources that are easily measurable (commits, builds, etc..).
    • Number of third party repositories or projects that decide to target Fedora Server.

    This last point may be a stretch goal – admittedly it will require some time before this becomes really measurable, but in a 2-year time frame this kind of adoption must show up.

    About this Document


    Contributors to this document include:

    • Stephen Gallagher
    • Jim Perrin
    • David Strauss
    • Truong Anh. Tuan
    • Máirín Duffy
    • Kevin Fenzi
    • Miloslav Trmač
    • Simo Sorce
    • Adam Williamson
    • Joe Brockmeier

Fedora Server PRD Draft and call for participation

The first deliverable that the Fedora Server Working Group was tasked with was the production of a Product Requirements Document. This document is intended to provide a high-level view of the goals and primary deliverables of the Fedora Server distribution. A great deal of discussion has gone on during the weekly Working Group meetings as well as on the mailing list.

At this time, the deadline for the delivery of the PRD is rapidly approaching. Originally it was due to be delivered for ratification on Monday, January 13th, but at the FESCo meeting on Wednesday, it was agreed to delay this deadline by a single week. The primary reason for this delay was so that the Fedora Cloud and Fedora Server groups could have some last discussions about overlap and respective areas of responsibility.

This past Tuesday, we had an all-day PRD hackfest in IRC and have come up with a fairly strong draft. It is not yet complete (notably, there remains a FIXME under “Misc. Concerns” and some ambiguity around the Use Cases), but I believe that it is close enough to its final form (as envisioned by those people that have contributed to it), that we should expose this document to the wider world and ask for input before submitting it to the Fedora Engineering Steering Committee and Fedora Advisory Board a week from Monday.

Please read through the PRD draft and provide feedback of any sort. If you see that we have missed or misrepresented any of our statements, we would very much like to hear this soon.

Fedora Server Working Group: Dec 3 Meeting

Fedora Server Working Group Logo

We had the sixth meeting of the Fedora Server Working Group last week.

How to Get Involved

Here are the various resources now affiliated with the Server Working Group that you can use to follow along or participate:

Meeting Minutes


Our agenda was set ahead of time on the mailing list after an active discussion on the call for agenda post.

  • Adam Williamson Confirmation
  • Discuss Role Implementation
  • PRD (we didn’t get to this topic)

Members Present

  • sgallagh
  • nirik
  • mizmo
  • simo
  • adamw
  • mitr
  • Evolution
  • davidstrauss

Adam Williamson Confirmation

As discussed at last week’s meeting, sgallagh approached the Fedora QA team to seek a nomination for a new Server working group member to fill Viking-Ice’s vacated spot. Adam Williamson volunteered to fill the vacated seat. This was supported all of the by the seven present voting members with no disagreement, so adamw is now the newest Server working group member.

Discuss Role Implementation

This conversation was very dense and meandering, and after spending a couple of hours trying to turn it into a useful readable dialog I decided it’s going to be more valuable to just summarize it up, point-by-point. That summary follows.


stefw and andreasn participated in this discussion from the perspective of the Cockpit project – a server console UI. They have been thinking about “ways to make more advanced server roles (sorta like some of the wizards in windows) discoverable and accessible via cockpit. [To] have a nice way to configure one of your servers as a Freeipa server, for example, is a goal of ours. Or to have a way to configure a server as a monitoring server.”

Server Role APIs

There was some discussion about designing an API – stefw cautioned the group against designing a generic server role API from the bottom up and instead recommended looking at how people should interact with it first.

sgallagh noted that OpenLMI could be a good abstract API to use.

davidstrauss remarked that [cockpit?] seems to map to systemd and thus it isn’t generic.

mitr pointed out that based on server list discussions:

  • UI is important
  • Automated installations are at least equally important.
  • One possibility that was suggested is to have a single ‘configuration’ for a role and to be able to ‘(re)apply it,’ i.e. not an API to change a value, but to remove/read a role with a new configuration (retaining data, of course).”

simo is fine with enabling cockpit and OpenLMI, but isn’t sure it’s a good idea to have Fedora Server rely on them.


Some suggestions that were brought up:

  • Move most of the package and operational config to post-installation so we can better support remote tools. (davidstrauss)
  • We should do as little configuration as possible by default, with the option for “advanced” knobs post-deployment. (sgallagh)
  • Install all packages, install config, and bring up a ‘standard’ (I’m not talking static. I mean, ‘ask for the minimum information necessary to reasonably install) config of the service, and then allow users to tweak after the fact. The way that most boxed server vendors work. (sgallagh)
  • Without a lot of info from the user up front (nirik)
  • It’s a spectrum from: just install the needed packages and user goes from there all the way to: install all packages, install config and bring up a ‘standard’ config of the service… (nirik)
  • For most networking services, config generally needs to be delayed anyway because you need hostnames and sometimes IP addresses to be the final ones, anyway. A lot of software can’t be correctly tweaked after install easily. (simo)
  • My vision of a ‘default install’ for a server OS would be basically Fedora’s current minimal. (adamw)
  • I would like to see minimum be truly minimal though. Not minimal for basic. but absolute bare-bones. (Evolution)
  • We could work with package maintainers to just have the package provide whatever default ‘standard’ thing to tweak out of the box? (nirik)
  • Should we ask at OS install time or at firstboot time? (simo)
    • That’s the part we need to figure out and the part that I was talking about having an ‘API’ for… To configure it for “default” use. I’m not committing to a complete configuration API right now. (sgallagh)
  • Basic configuration should be able to be performed by the installer or a management console like cockpit. (sgallagh)
  • stefw is hoping that openlmi will provide the scripting and cockpit the UI, and cockpit can use OpenLMI as appropriate.
  • Scripts are complex, but using scripts to manage configuration sounds like debconf, and it seems debconf is a model we’d like to avoid:
    • The scripts create a lot of maintenance overhead
    • It breaks automated config tools because of how its interactive configuration works
    • It starts things after installation, without using the config you wanted for initial startup.
    • It’s integrated with the packaging system. Package installation and configuration are two very separate steps.

Even with the bulleted list approach above, you can see how this was kind of all over the place. :)

Taking a Step Back

I tried to come up with a rough long-term vision for how server role installation and configuration would work based on the discussion:

“When you install a role, it comes prepackaged with a set of sane defaults which you can configure post-install.”

davidstrauss noted that it’s already 95% of the case. I asked how is this vision solving anything, and stefw said that sane defaults don’t come with many packages. “Fixing up software and packages so that they’re ‘roleable’ is a big part of this,” he also said.

adamw also suggested,

“It sounds like we’re discussing some sort of distro-owned configuration/wizard/helper layer between the default configuration we set in packages and whatever configuration guides/tools the upstream software provides, which I thought was a business Fedora had been trying to get out of.”

sgallagh explained,

“We want to behave more like Windows Server does here: If someone tells Server Manager that ‘that machine over there is a domain controller,’ it gives you a wizard to set up the basic information and then later allows you to tweak it further.”

I brought up that I recently tried to set up postfix and it was a bit of a nightmare, and I wasn’t sure how you could configure it to run out-of-the-box when so much of the setup was specific to a given network / domain.

“You cannot solve that problem w/o building something like kolab,” said simo. “I do not want to get Fedora in the business of building in tightly integrated roles, because we will fail.”

simo, mitr, and sgallagh agreed that, for at least the short term, packaging only simple roles or existing integrated roles is something we should focus on, and we shouldn’t try to integrate non-existing roles.

davidstrauss suggested using containers to allow tight integration between daemons. “[It's] easy to run into conflicts between how one ‘app’ wants DNS configured and another one does.” mitr disagreed with the container approach, though, “containers do pretty much nothing valuable to the user in this aspect – isolation is an aspect of reliability, not an end-user feature.”

“I’m actually saying with the container stance that we should punt on the problem,” said davidstrauss. “I don’t think we’re equipped to do the integrated recipes.”

There was some discussion about implementation – OpenShift gear recipes and docker were brought up.

“I’m very anti-NIH. Proudly invented elsewhere should be a goal of ours,” said davidstrauss, and the whole group appeared to agree.

Vision Statement

Post-meeting, I expanded the vision statement for role installation based on the above discussion,

  • Roles will be maintainable objects in a repo managed with Fedora
    maintainer standards, integrated & coherent for things like email
    server, groupware, etc.
  • Users can select roles to add to a system, and would be provided with
    a (wordpress first-run like) wizard for some initial information and
    they’ll be given a working server using sane defaults. Optionally, the
    user should also be able to deploy the role in a mass deployment without
    having to interact with the wizard.
  • The configuration should be able to be modified post-install (but not
    necessarily via GUI right away; it’s hard to commit to this.)

How to move forward

Generally everyone seemed to suggest that we need to choose 1-3 roles for the first release, and learn the best approach from exploring them.

simo also suggested that someone needs to take a stab at proposing a clear view and provide a well-thought-out example. sgallagh volunteered to do this using FreeIPA as the example, which he has already posted.

Meetbot Minutes

Here is the official meetbot meeting minutes with links to the full raw log:

Fedora Server Working Group: Nov 26 Meeting

Fedora Server Working Group Logo

Today was the fifth meeting of the Fedora Server Working Group!

How to Get Involved

Here are the various resources now affiliated with the Server Working Group that you can use to follow along or participate:

Meeting Minutes


Our agenda was set ahead of time on the mailing list after an active discussion on the call for agenda post.

  • Select a new member of the Working Group
  • Personas

Members Present

  • sgallagh
  • nirik
  • mitr
  • simo
  • mizmo
  • davidstrauss
  • tuanta
  • Evolution

Working Group Member Selection

Jóhann (Viking-Ice) decided to step down from the server working group. So the server working group needs to select a new working group member. As mitr pointed out, our charter says we should choose from “from the active Fedora Server community” but the group hasn’t been around long enough to build up an active community.

Since Jóhann represented the QA community in Fedora, we agreed on the following:

  • We will ask the QA community to recommend someone from their ranks.
  • If no one from QA steps forward, we will put out a general call, particularly to those who previously volunteered.


Most of the meeting time was spent on discussing personas. I (mizmo) gave a status update on where the personas are at right now:

  • We have a working personas document available at: It’s based on the personas nirik suggested at last week’s meeting.
  • We have one sysadmin interview completed to help inform the personas. mizmo and sgallagh interviewed kdetony last week. He is a sysadmin for a large company. (The interview notes are now available in the wiki.) mizmo will interview davidstrauss too when he returns to the US.
  • sgallagh and mizmo basically put together a FAQ on personas as well (I just put it up on the wiki now.)

A quick review

Evolution missed the last meeting, so we filled him in on the point of personas and how we’d go about using them. I added our answers to the persona FAQ so if you are curious about these things please check that out.

“I like the general concept,” said Evolution, “so long as we don’t get caught up in arbitrarily pigeon-holing things because of this.”

“We all agree on that score,” sgallagh assured him.

Persona order?

Nirik also had a great question about whether or not personas have an order to them; they can – for example, we can define a couple of primary personas, have secondary personas, and maybe anti-personas (we would explicitly not cater to the needs of the anti-personas, but could define them for clarity.)

The personas from the first draft

So next I walked folks through the three personas we have:

  • Senior sysadmin / nonprofit org / she’s classified as ‘MacGuyver’ because they don’t have a lot of resources but try to get a lot done
  • Server app developer / freelancer / (no nickname yet)
  • Junior sys admin / huge company / lots of resources available (no nickname for him yet either)

What does junior mean, and another persona suggestion

davidstrauss asked, “What makes the third one ‘junior?’”

“He might not have as much decision-making power,” I (mizmo) replied, “He’s not the team lead or anything like that. so he may be stuck with decisions made over his head.”

“It seems strange to couple abundant company resources with the ‘junior’ guy,” davidstrauss responded.

I replied, “Well, not necessarily. E.g., if he needs more hardware he probably won’t have a hard time getting it, but he’s not going to be able to change the bureacracy at his org.”

“Well, I guess I’m the contrast to that persona, anyway. We have lots of resources, and I’m CTO,” said davidstrauss.

“I think you would be an excellent additional persona,” I replied. “I would probably nickname your persona ‘decision maker.’”

davidstrauss answered, “I don’t really care if it’s me, but we need someone with resources + power.”

“Yes, a decision maker is an important case,” mitr added.

Going meta

sgallagh suggested that we add a persona for folks creating a new server role for Fedora Server. This would be a kind of meta persona – a person who contributes to Fedora by putting together a role for the server platform product. We’ll add it as a secondary persona.

The question this persona might help answer, as posed by davidstrauss: “How does one create a service that plugs nicely into Fedora?”

Developer vs. DevOps

“I do agree we need a developer persona (particularly for the requirements on languages and deployment),” mitr pointed out. “Can we clarify that Server is not the OS one uses to actually develop the application (i.e. run Eclipse / ask on stackoverflow …) – that should be Workstation [... which requires inter-WG agreement on the languages and deployment mechanism)."

nirik added, "Yes, but we shouldn't ignore that case... they need to install server software to test, etc."

I (mizmo) took the action item to modify the developer persona accordingly.

I also asked, "Should we have a devops persona too in addition to the developer?"

"I thought that was what your developer persona was, there," sgallagh responded.

mitr asked, "What would be unique about devops? Ability to deploy and to monitor should already be there."

"We could make him devops then," I said, "I think Ijust need more info about how the guy would work. I don't know enough about it. E.g., does a devops person get paged at all hours when there's an outage?"

mitr said, "I'd slightly prefer not restricting ourselves to devops (multiple platforms, and speed to deploy are equally important when deploying a 10-year old J2EE application), but I don't feel strongly about it."

"The primary difference with DevOps," sgallagh explained, "is the concept of 'Mean time to recovery' is more important than 'Uptime.'"

"Does devops necessarily mean a smaller organization?" I asked. "Because I don't know how you'd have them combined into one role for a huge app like the one I'm thinking of."

Simo responded, "Well, devops people would say no, but in reality I see real devops only in startups. :)"

"Devops kind of captures nicely "the Ruby problem" (sorry, I'm unfairly picking on a single language for simplicity)," mitr added.

I asked, "What's the ruby problem?"

"The ruby problem is mainly 'the bundling problem' rehashed," sgallagh answered. "Except with a community that has no stake in backwards-compatibility (in the general case.)"

"What sgallagh said, + the culture that 'that's how things are done.'" added mitr.

To confirm my understanding, I asked, "So with ruby you need devops, because the platform isn't backwards compatible, so you need a developer to port things forward if something goes wrong because of a bug in the platform or whatnot?"

"Precisely," sgallagh answered.

simo added, "It is not just ruby."

"Right, ruby is just a representative example," sgallagh said. "This is true of a lot of Java shops too."

"It is also: oh cloud provider X we use is going to change API in 10 days," explained simo. "Need to develop a different connector for the production servers ASAP. Language really doesn't matter"

"And Google APIs..." added sgallagh. "So it's really about resiliency not just of the service but also the deployment itself."

Simo explained further, "It's about people using *new* platforms that keep changing and shifting all the time. They use so much of *everything* it is all in costant motion."

"Well, it's also about a perception change," sgallagh added. "That absolute stability is not as important as rapid recovery."

davidstrauss said, "It's part of the perception change I'm referring to in the migrations vs. support duration [thread]. How easy is it to keep the app on a supported foundation? *versus* How long can the foundation we’ve deployed to remain supported?”

“It is a lot about automation,” said Simo, “because devops will redeploy a lot. A lot more than traditional ops used to.”

Nirik also illustrated the point, “‘Thing x is hard and we only do it once a year, so, lets do it 10x/day now and fix all the problems so that we can keep going.’”

“Right, the devops folks pretty much live in the ‘livestock’ side of things, except possibly for their hypervisors,” added sgallagh, referring back to the analogy that some servers are treated as expendable (like livestock), and some
are given names and fixed up when they get broken and are kept around long-term (like pets.)

“They concentrate on the changing pieces but absolutely rely on the stable infra as well,” simo said.

I asked, to confirm, “So devops people are more automated, the traditional people are more ‘omg don’t break it?’”

“That’s a pretty fair analogy,” replied sgallagh.

So we concluded that I’d make the developer persona more ‘devoppy,’ now that I had a better understanding of what that meant.

Where does Server end and Cloud begin?

One thing that came up at this point in the conversation was the line between the Cloud product and the Server product.

“When it comes to server and cloud usage, I want Fedora to leapfrog the current popular choices philosophically,” said davidstrauss. “Our competition is, in some ways, CoreOS more than Ubuntu.”

nirik added, “I think we can add a lot of value in standardizing/best practices…”

“Let’s also not get in the Cloud group’s way too much either,” sgallagh pointed out. “I think we probably want to be focusing on the infrastructure to support their product. Fedora Cloud is likely to be Fedora’s answer to CoreOS.”

“I’d generally prefer if a Cloud application and Server application were bit-for-bit the same (but it hasn’t been discussed yet,)” mitr added. davidstrauss agreed.

“To be honest, I see less and less separation between server and cloud product,” said simo in response to sgallagh. “I wonder if they really will be that different if we keep pushing hypervisor stuff that way.”

“I always thought cloud os would care about guests more than bare metal,” simo said.

Evolution responded, “There’s a large amount of crossover in the two.”

“But the guests are often-times running the server bits we’re interested in,” replied Evolution.

simo in response said, “I wonder if it really make sense to have separate stuff in the end.”

“We deploy Fedora to bare metal as a container host,” davidstrauss explained. “Once companies like GitHub, Etsy, etc. grow to a certain point, they often substantially leave ‘cloud’ for bare metal. And once they leave cloud, they either deploy to hypervisors on their own hardware or something like the “guest images” to bare metal directly.”

sgallagh said, “Whereas our position is to deliver sturdy infrastructure both for traditional roles and to support Cloud deployments.”

simo replied to davidstrauss, “True, many company have mixed environments, with in house clouds + overflow on public clouds after some time.”

“Some companies opt to run their own clouds. Others move to PXE deployments and similar,” responded davidstrauss.

Traditional Developer Persona

“Did anybody have a chance to read through some of the stuff on the developer persona? I kind of really just made it up but I don’t know if I was getting at the wrong thing,” I asked. “E.g., under frustrations: ‘Spending cycles porting code to an endless array of platforms – it’s time-consuming and uninteresting work,’ but I guess a devop wouldn’t do that?”

sgallagh answered, “No, devops would pick a minimalist platform and just VM/Containerize it.”

I asked further, “Do we care about devs/admins who would need to do porting work like that on a server?”

“Traditional app developers will need to be served, yes,” sgallagh answered, “The Oracle DBs and SAPs of the world, for example.”

I asked for more examples of traditional deployments, and got the following:

  • Domain controller
  • DNS server
  • Email infrastructure
  • File server
  • Storage cluster
  • Office proxy
  • Office DNS and web proxy

Personas and timelines

“We probably should try and finish personas in not too long… given our short timeframe to come up with deliverables/document,” said nirik.

“I think we’re pretty close,” I pointed out. “I didn’t see a whole lot of contention here about the ones we have so fair, just some refinements / expansion.”

As sgallagh proposed, I’ll try to have the personas document ready to be voted on December 10.

Meetbot Minutes

Here is the official meetbot meeting minutes with links to the full raw log: