Date
1 - 9 of 9
Need for offline binary configuration
|
Venkata ramana gollamudi <ramana.gollamudi@...>
Poky allows to build custom Linux for you, but we have cases where the post build customization is required, like user-addition, network configuration, service control. Even selecting the required packages can be a post build activity.
The current model requires the image to be rebuilt to support these configuration. Offline Configuration tool (OCT), which allows a binary image customization before making a final target image. This case will be more evident in larger companies, where platform teams, product teams , application teams are distributed and Linux build from source will be owned and lab tested by a single team, like platform team. Other teams just configure to use it for product variants from same platform build. Detailed use cases can be found in enhancement bug:3252 OCT should work on the binary pool of compiled packages generated from poky. The basic operations that can be supported includes a) Select/deselect required packages from pool of binary packages into final target image. b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image. c) Binary level Offline configuration can includes Configure the users/passwords Configure the network Configure the host name Select the services to be started by default Security related configuration Generate initrd in ramfs/ext3/... format etc.. Considering the methods to support these in our current yocto model, following changes can be done. 1) HOB can be the tool which can be extended to support these Poky can generate a binary package pool as one if its output and Hob can work on this package pool to select packages, configure and generate image. So HOB can support opening HOB in Binary(or OCT) mode i.e., without build options but only with binary package selection. Configuration GUI needs to be added to HOB. Note:HOB+OCT is together or separate, needs a bit more thought and overall organization as they will be intended for different users. 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure. I think it is sufficient to keep the minimal binaries from sstate-cache which are required to execute image.bbclass, do_rootfs task to generate image. 3) Along with specific configuration UI implementation, a generic configuration model similar to kernel kconfig and menuconfig can be considered, in cases where more detailed offline configurations is required like detailed security configuration. Regards, Ramana |
|
|
|
Bruce Ashfield <bruce.ashfield@...>
On 12-11-20 10:09 AM, Venkata ramana gollamudi wrote:
Poky allows to build custom Linux for you, but we have cases where the post build customization is required, like user-addition, network configuration, service control. Even selecting the required packages can be a post build activity.Is there some overlap between this point and the other ongoing discussions about image construction, deployment and package management ? i.e. this thread: [OE-core] RFC: OE-Core image creation and deployment http://lists.linuxtogo.org/pipermail/openembedded-core/2012-July/026938.html These may already be coordinated, but I do see some common threads and it would be nice to make sure everything will work together and that we aren't duplicating effort! Cheers, Bruce 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure. |
|
|
|
David Stewart
There are a ton of interesting interactions here with other efforts -
please review and comment. On 11/20/12 7:09 AM, "Venkata ramana gollamudi" <ramana.gollamudi@...> wrote: Poky allows to build custom Linux for you, but we have cases where theI can certainly see how something like this would be a valuable addition to the workflow, particularly at large organizations. Of course, you could conceive of all of these things being configured in a layer as well, but would require another bitbake run. I think you are suggesting something separate from bitbake, and a lot more graphical / interactive.
|
|
|
|
Venkata ramana gollamudi <ramana.gollamudi@...>
Reply inline
toggle quoted message
Show quoted text
-----Original Message-----Bruce, Thanks for the information. After your reply, I have gone through the discussions and agree that binary pool is in similar lines. Great to see that the realization happening in yocto1.4. Understood that package-feed can be used to generate the target image. Is there any RFC/mail/wiki available explaining how the configuration(like fstype) during image generation will be realized? I am looking for post build configuration tool, which allows the product team users also to configure users, network, services etc . Image type, file system and Partition configuration can be one of them. I expect the product team users who configures image and generates target image, will have a little or no knowledge of bitbake, also needs easy installation and less dependencies. Can look in this context, how HOB will fit into this scenario or needs a new tool. Point 2, No longer applicable as package-feed is a binary pool.2) Binary package pool can be a minimal/partial sstate-cache, ascomplete sstate-cache is quite big and not required for product teams Point 3, still can be thought about.3) Along with specific configuration UI implementation, a genericconfiguration model similar to kernel kconfig and menuconfig can be
|
|
|
|
Bruce Ashfield <bruce.ashfield@...>
On 12-11-21 11:29 AM, Venkata ramana gollamudi wrote:
Reply inlineNot that I know of. It is still under design last I heard, but MarkH is the-----Original Message-----Bruce, Thanks for the information. After your reply, I have gone through the discussions and agree that binary pool is in similar lines. Great to see that the realization happening in yocto1.4. person to provide the details. He's out of the office at the moment, but I'm sure that when he is back he can provide plenty of information. Agreed. I see this as something to start with, since it doesn't overlap with the other efforts (that I know of), and when I first read your email I thought it was the main focus. When you continued into image creation and package selection, that's when I noted the overlap. Image type, file system and Partition configuration can be one of them.Keeping the number of tools low is a good thing, so hopefully it can fit within the existing options. There have been other menuconfig efforts in the past (that I've heardPoint 2, No longer applicable as package-feed is a binary pool.2) Binary package pool can be a minimal/partial sstate-cache, ascomplete sstate-cache is quite big and not required for product teams about, but not had direct involvement), so doing some research in this area would be appropriate as well. cheers, Bruce
|
|
|
|
Venkata ramana gollamudi <ramana.gollamudi@...>
Looking for comments. Thanks
________________________________________ From: Bruce Ashfield [bruce.ashfield@...] Sent: Thursday, November 22, 2012 12:59 AM To: Venkata ramana gollamudi Cc: yocto@...; Sanil kumar; Hatle, Mark; 'dvhart@...' Subject: Re: [yocto] Need for offline binary configuration On 12-11-21 11:29 AM, Venkata ramana gollamudi wrote: Reply inlineNot that I know of. It is still under design last I heard, but MarkH is the-----Original Message-----Bruce, Thanks for the information. After your reply, I have gone through the discussions and agree that binary pool is in similar lines. Great to see that the realization happening in yocto1.4. person to provide the details. He's out of the office at the moment, but I'm sure that when he is back he can provide plenty of information. Agreed. I see this as something to start with, since it doesn't overlap with the other efforts (that I know of), and when I first read your email I thought it was the main focus. When you continued into image creation and package selection, that's when I noted the overlap. Image type, file system and Partition configuration can be one of them.Keeping the number of tools low is a good thing, so hopefully it can fit within the existing options. There have been other menuconfig efforts in the past (that I've heardPoint 2, No longer applicable as package-feed is a binary pool.2) Binary package pool can be a minimal/partial sstate-cache, ascomplete sstate-cache is quite big and not required for product teams about, but not had direct involvement), so doing some research in this area would be appropriate as well. cheers, Bruce
|
|
|
|
Venkata ramana gollamudi <ramana.gollamudi@...>
Added few more thoughts
Looking for possible options to implement Offline Configuration Tool requirement. please check and comment on it. Configuration flow: a) Task do_createconfig will generate data to configure. This serves as input for configuration UI in OCT. b) OCT UI will read that config data(from step(a)) to show it in GUI. User configures using GUI inputs. Stores the user OCT configuration to OCT config file. (Note user can directly modify the OCT config file, if User is familier with it) c) preinst, postinst, prerm, postrm, or a do_applyconfig(without bb files reparse), can read OCT config file content (using a library) and apply necessary changes. Now we can look into possibility of using a specific configuration UI implementation or a generic configuration model or combination of both. 1) Using a specific configuration GUI implementation: do_createconfig generates the configuration information, in format like datafile/xml and OCT reads this config data and provides a GUI to modify or configure the features. Pros: Detailed and complex GUIs can be supported which will be user friendly. Ex: User management, network management etc. Cons: As UI is static, it has to be modified to support new configuration requirements and UI modifications are generally complex. 2) A generic configuration model do_createconfig generates configuration information with presentation information included (like kconfig) and OCT reads this config information and provides a generic UI implementation to modify or config the data like menuconfig. Pros: - Generic and dynamic UI allowing the config menus to be added from poky layers providing platform for configuration. - New features can add new menus configuration options. Feature/package enable and disable can control the configurtion menus. - So configuration is shown only if feature/package is present/selected. Cons: - Generic method of implementing can be complex, specifically for features where configuration is complex ex: User management, network configuration. - Custom or specific UI will be more user friendly and gives more control to UI designer. 3) Combination of both Complex features requiring more UI control can implement those features using the method(1) and also providing the plaform for generic UI implementation. OCT displays both UI configurations. Ex: Static Image level features like User management, network config can be implemented using method(1) Service configuration, file system type, hist size configure etc can be implemented using method(2). Pros: Gives control to project to select the Complex and simple UIs. Cons: Still needs a OCT UI change if method(1) feature need to be changed. Drawing a line between complex and simple UI features may not very straight forward. Adding dynamic Complex UI is not possible to be added from poky layers. Decision to provide a generic configuration model is mainly based on how much we expect the configuration area to grow along with layers. Image level features like user management, network management generally are not dynamic features, so having a specific/custom UI implementation looks fine for such features. Regards, Ramana |
|
|
|
Mark Hatle <mark.hatle@...>
My apologies for the late response. See below, I'll follow up on your second email as well...
On 11/20/12 9:09 AM, Venkata ramana gollamudi wrote: Poky allows to build custom Linux for you, but we have cases where the postOne of the objectives that we are currently working toward is being able to start with the package feed (doesn't matter if it's rpm, deb or ipk files), generate a rootfs, configure the rootfs and then deploy the rootfs into an image. The above should be able to be done on both a cross-configured environment or on the target via a traditional installer like mechanism. The current model requires the image to be rebuilt to support these configuration.The current model can avoid many of the tasks associated with the build by using sstate-cache, but I agree that is not optimal for users who want to start with a known set of binaries and then tailor from that point. (This includes users expecting a more traditional Linux install experience or users who need immutable software from an upstream vendor.) Offline Configuration tool (OCT), which allows a binary image customizationThis was factored into the work we are doing. I know many companies have an outside OSV that provides the build environment, support, etc.. and they feed into an Operating Systems group (or similar). This group then prepares the software according to the corporate guidelines (license restrictions, quality assurance, etc) and repackages it for consumption by others in the organization. Often time the output of the Operating System group is either a blessed source based system -- or a set of binaries that are blessed as being in conformance with that companies guidelines. This process repeats through the platform group and even individual projects -- with the eventual output being a product image including the blessed binaries, custom applications and configurations -- and a matching SDK. Detailed use cases can be found in enhancement bug:3252For the next section, I'm going to assume the pool of binary packages is what we refer to as the package feeds in OpenEmbedded. This is simple a collection of binary packages that have been built from the build system that can be used to construct the rootfs. Each package feed is constructed based on a single "distribution" configuration file which sets the policies for the packages in the feed and distribution wide options, such as selinux, pam, etc. a) Select/deselect required packages from pool of binary packages into final target image.The above is definitely considered in the work we've been talking about. This will require software that will take a list of the available items in the feed, present the user/developer with a way to enable the components they want on the target. (Note, there is an inherit requirement here that dependencies also be met, so things the user/developer may not have selected will either have to be installed to meet dependencies or will have to generate an error that dependencies can't be met.) b) Provision to select external binary packages like ADT compiled applications as input and add them to final target image.As long as the external package is in the same format as the rest of the feeds, this should be easy to do. And I certainly expect that it will be needed. c) Binary level Offline configuration can includesThe above are things we've talked about, but I don't believe there have been any formal plans to implement them at this time. The components necessary for the binary package based install -> rootfs generation have taken up most of our planning time at this point. One thing I would love to see is a set of standard components (or a framework) that could be used for this type of configuration. But I believe it's very important that the configuration work both within the OpenEmbedded environment and externally in the "installer" like environment. Security related configurationThis is something that I don't believe has really been addressed at all, but like the above we need a way to implement it. Generate initrd in ramfs/ext3/... formatWe are already considering how to go from rootfs to deployable image. initrd just happens to be one of many potential image types. We've specified hard disk image (partition map, bootloader, individual partitions that comprise one or more rootfs). A flash based image -- instruction for flashing, and individual flash 'partitions' that will work similarly to the hard drive. A CD/DVD/USB based bootable system. And finally as a component of these an initrd rootfs that can be used as a component of the others are separately to instantiate the system before handing off control to one of the other types. We have a huge advantage in OE that we can generate multiple rootfses from a given distribution configuration, we just don't currently have the tooling to put those rootfses together into the various images. I know Darren Hart at Intel is working on some of the ground work for the hard disk style images. Considering the methods to support these in our current yocto model, following changes can be done.The first thing we would like to see is a -single- rootfs creation model for all of the modes. Build system, cross-install, and target-based-install. The way that I'm approaching this right now is by attempting to abstract out just the rootfs generation capabilities within the current build system. The intention is that you will be able to access interfaces to get a list of whats available to be installed, (what may be already installed,) a way to pass what to install and where to the environment. Then the same code will be executed to perform these actions. The actions will be designed to be package type agnostic as well. Early R&D work is going on there, just to see if I can come up with something reasonable. 1) HOB can be the tool which can be extended to support theseI'm not sure HOB itself is the right tool for this, but there are certainly facets of HOB that would be needed in this kind of tool. The package selection, etc. 2) Binary package pool can be a minimal/partial sstate-cache, as complete sstate-cache is quite big and not required for product teams as they are not expected to build but just need to select and configure.There are multiple steps of binary package pool, each I think are valid. For someone wanting a source based system, with prebuilt binaries that can be locally modified -- sstate-cache is by far the best way to do this. For an applications group that needs to generate images and SDKs, but is never allowed to use anything but the corporate blessed binary packages, a different mechanism (package based) is likely what is needed. There is a lot of area though where either method may be needed (or desired). My experience shows people familiar with traditional workstation based systems prefer the binary RPM approach, while hard-core embedded developers prefer source based systems. The reality is the world has fewer hard-code embedded developers and more folks used to the traditional environments, add to that the capabilities of today's embedded systems and it makes sense for a lot of people to start with blessed binaries, tailored for embedded systems, and simple add their integration of applications and configurations after. 3) Along with specific configuration UI implementation, a generic configuration model similar to kernel kconfig and menuconfig can be considered, in cases where more detailed offline configurations is required like detailed security configuration.This is absolutely need, but I think we need to get the general rootfs generation working and figured out before we spend too much time on this. However, if this is a high priority for you, it may make sense to start working on configuration. This is something that is definitely needed, and I don't believe anyone is actively working on. --Mark Regards, |
|
|
|
Mark Hatle <mark.hatle@...>
On 11/29/12 5:09 AM, Venkata ramana gollamudi wrote:
Added few more thoughtsUsing tasks to control the configuration elements will require the build system and recipes. This is really good for the existing workflow, but doesn't help with the blessed binaries situation. What I'd love to see (but don't expect) is a way for the binary packages to document he configuration files and options in a way that the UI can load this information, give the user simple check-box/drop down style configuration -- or simply a text editor with the end result something that gets back to the target environment. b) OCT UI will read that config data(from step(a)) to show it in GUI. User configures using GUI inputs. Stores the user OCT configuration to OCT config file. (Note user can directly modify the OCT config file, if User is familier with it)I'm not sure pre/post is the right place to do these changes. Typically I've seen the default configuration files marked as such within the package system and then the modifications are made externally. Sometimes they're made in ancillary packages or just made raw on the disk. Most package management systems I'm aware of can handle this just fine. My concern with modifying the defaults is that the feed become project specific, and again it requires input during the build process as opposed to the install process. Now we can look into possibility of using a specific configuration UI implementation or a generic configuration model or combination of both.Having something like a do_createconfig (or even a do_describeconfig) to generate a file that is later made available to the UI tool is my suggestion. Then the input/output of the tool ends up in the rootfs directly. (The methods to get to the rootfs should be either a specific path, a URL for get/put operations, or a method like tcf/ssh to copy the item to a running target. Either a custom per-package model or a generic model may be needed. I don't know what the best approach is -- however I know previous attempts to do this have generally resulted in more 'custom' model then generic + config file model. But in general there seem to be three main types of configuration files that I've seen: Column/format based -- i.e. /etc/fstab, /etc/exports Key/Value based -- things like Key=Value, or Key:\nValue, etc.. XML-like -- Highly formatted configuration files like apache. --Mark 2) A generic configuration model |
|
|