Build a Debian package against Debian 8.0 using Download On Demand (DoD) service
In the previous post Open Build Service software architecture has been overviewed. In the current blog post, a tutorial on setting up a package build with OBS from Debian packages is presented.
Steps:
- Generate a test environment by creating Stretch/SID VM
- Enable experimental repository
- Install OBS server, api, worker and osc CLI packages
- Ensure all OBS services are running
- Create an OBS project for Download on Demand (DoD)
- Create an OBS project linked to DoD
- Adding a package to the project
- Troubleshooting OBS
Generate a test environment by creating Stretch/SID VM
Really, use whatever suits you best, but please create an untrusted test environment for this one.
In the current tutorial it assumes “$hostname” is “stretch”, which should be stretch or sid suite.
Be aware that copy & paste configuration files from current post might lead you into broken characters (i.e. “).
Debian Stretch weekly netinst CD
Enable experimental repository
# echo "deb http://httpredir.debian.org/debian experimental main" >> /etc/apt/sources.list.d/experimental.list # apt-get update
Install and setup OBS server, api, worker and osc CLI packages
# apt-get install obs-server obs-api obs-worker osc
In the install process mysql database is needed, therefore if mysql server is not setup, a password needs to be provided.
When OBS API database ‘obs-api‘ is created, we need to pick a password for it, provide “opensuse”. The ‘obs-api’ package will configure apache2 https webserver (creating a dummy certificate for “stretch”) to serve OBS webui.
Add “stretch” and “obs” aliases to “localhost” entry in your /etc/hosts file.
Enable worker by setting ENABLED=1 in /etc/default/obsworker
Try to connect to the web UI https://stretch/
Login into OBS webui, default login credentials: Admin/opensuse).
From command line tool, try to list projects in OBS
$ osc -A https://stretch ls
Accept dummy certificate and provide credentials (defaults: Admin/opensuse)
If the install proceeds as expected follow to the next step.
Ensure all OBS services are running
# backend services obsrun 813 0.0 0.9 104960 20448 ? Ss 08:33 0:03 /usr/bin/perl -w /usr/lib/obs/server/bs_dodup obsrun 815 0.0 1.5 157512 31940 ? Ss 08:33 0:07 /usr/bin/perl -w /usr/lib/obs/server/bs_repserver obsrun 1295 0.0 1.6 157644 32960 ? S 08:34 0:07 \_ /usr/bin/perl -w /usr/lib/obs/server/bs_repserver obsrun 816 0.0 1.8 167972 38600 ? Ss 08:33 0:08 /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver obsrun 1296 0.0 1.8 168100 38864 ? S 08:34 0:09 \_ /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver memcache 817 0.0 0.6 346964 12872 ? Ssl 08:33 0:11 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 obsrun 818 0.1 0.5 78548 11884 ? Ss 08:33 0:41 /usr/bin/perl -w /usr/lib/obs/server/bs_dispatch obsserv+ 819 0.0 0.3 77516 7196 ? Ss 08:33 0:05 /usr/bin/perl -w /usr/lib/obs/server/bs_service mysql 851 0.0 0.0 4284 1324 ? Ss 08:33 0:00 /bin/sh /usr/bin/mysqld_safe mysql 1239 0.2 6.3 1010744 130104 ? Sl 08:33 1:31 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 # web services root 1452 0.0 0.1 110020 3968 ? Ss 08:34 0:01 /usr/sbin/apache2 -k start root 1454 0.0 0.1 435992 3496 ? Ssl 08:34 0:00 \_ Passenger watchdog root 1460 0.3 0.2 651044 5188 ? Sl 08:34 1:46 | \_ Passenger core nobody 1465 0.0 0.1 444572 3312 ? Sl 08:34 0:00 | \_ Passenger ust-router www-data 1476 0.0 0.1 855892 2608 ? Sl 08:34 0:09 \_ /usr/sbin/apache2 -k start www-data 1477 0.0 0.1 856068 2880 ? Sl 08:34 0:09 \_ /usr/sbin/apache2 -k start www-data 1761 0.0 4.9 426868 102040 ? Sl 08:34 0:29 delayed_job.0 www-data 1767 0.0 4.8 425624 99888 ? Sl 08:34 0:30 delayed_job.1 www-data 1775 0.0 4.9 426516 101708 ? Sl 08:34 0:28 delayed_job.2 nobody 1788 0.0 5.7 496092 117480 ? Sl 08:34 0:03 Passenger RubyApp: /usr/share/obs/api nobody 1796 0.0 4.9 488888 102176 ? Sl 08:34 0:00 Passenger RubyApp: /usr/share/obs/api www-data 1814 0.0 4.5 282576 92376 ? Sl 08:34 0:22 delayed_job.1000 www-data 1829 0.0 4.4 282684 92228 ? Sl 08:34 0:22 delayed_job.1010 www-data 1841 0.0 4.5 282932 92536 ? Sl 08:34 0:22 delayed_job.1020 www-data 1855 0.0 4.9 427988 101492 ? Sl 08:34 0:29 delayed_job.1030 www-data 1865 0.2 5.0 492500 102964 ? Sl 08:34 1:09 clockworkd.clock www-data 1899 0.0 0.0 87100 1400 ? S 08:34 0:00 /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf www-data 1900 0.1 0.4 161620 8276 ? Sl 08:34 0:51 \_ /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf # OBS worker root 1604 0.0 0.0 28116 1492 ? Ss 08:34 0:00 SCREEN -m -d -c /srv/obs/run/worker/boot/screenrc root 1605 0.0 0.9 75424 18764 pts/0 Ss+ 08:34 0:06 \_ /usr/bin/perl -w ./bs_worker --hardstatus --root /srv/obs/worker/root_1 --statedir /srv/obs/run/worker/1 --id stretch:1 --reposerver http://obs:5252 --jobs 1
Create an OBS project for Download on Demand (DoD)
Create a meta project file:
$ osc -A https://stretch:443 meta prj Debian:8 -e
<project name=”Debian:8″>
<title>Debian 8 DoD</title>
<description>Debian 8 DoD</description>
<person userid=”Admin” role=”maintainer”/>
<repository name=”main”>
<download arch=”x86_64″ url=”http://deb.debian.org/debian/jessie/main” repotype=”deb”/>
<arch>x86_64</arch>
</repository>
</project>
Visit webUI to check project configuration
Create a meta project configuration file:
$ osc -A https://stretch:443 meta prjconf Debian:8 -e
Add the following file, as found at build.opensuse.org
Repotype: debian # create initial user Preinstall: base-passwd Preinstall: user-setup # required for preinstall images Preinstall: perl # preinstall essentials + dependencies Preinstall: base-files base-passwd bash bsdutils coreutils dash debconf Preinstall: debianutils diffutils dpkg e2fslibs e2fsprogs findutils gawk Preinstall: gcc-4.9-base grep gzip hostname initscripts insserv libacl1 Preinstall: libattr1 libblkid1 libbz2-1.0 libc-bin libc6 libcomerr2 libdb5.3 Preinstall: libgcc1 liblzma5 libmount1 libncurses5 libpam-modules Preinstall: libpcre3 libsmartcols1 Preinstall: libpam-modules-bin libpam-runtime libpam0g libreadline6 Preinstall: libselinux1 libsemanage-common libsemanage1 libsepol1 libsigsegv2 Preinstall: libslang2 libss2 libtinfo5 libustr-1.0-1 libuuid1 login lsb-base Preinstall: mount multiarch-support ncurses-base ncurses-bin passwd perl-base Preinstall: readline-common sed sensible-utils sysv-rc sysvinit sysvinit-utils Preinstall: tar tzdata util-linux zlib1g Runscripts: base-passwd user-setup base-files gawk VMinstall: libdevmapper1.02.1 Order: user-setup:base-files # Essential packages (this should also pull the dependencies) Support: base-files base-passwd bash bsdutils coreutils dash debianutils Support: diffutils dpkg e2fsprogs findutils grep gzip hostname libc-bin Support: login mount ncurses-base ncurses-bin perl-base sed sysvinit Support: sysvinit-utils tar util-linux # Build-essentials Required: build-essential Prefer: build-essential:make # build script needs fakeroot Support: fakeroot # lintian support would be nice, but breaks too much atm #Support: lintian # helper tools in the chroot Support: less kmod net-tools procps psmisc strace vim # everything below same as for Debian:6.0 (apart from the version macros ofc) # circular dependendencies in openjdk stack Order: openjdk-6-jre-lib:openjdk-6-jre-headless Order: openjdk-6-jre-headless:ca-certificates-java Keep: binutils cpp cracklib file findutils gawk gcc gcc-ada gcc-c++ Keep: gzip libada libstdc++ libunwind Keep: libunwind-devel libzio make mktemp pam-devel pam-modules Keep: patch perl rcs timezone Prefer: cvs libesd0 libfam0 libfam-dev expect Prefer: gawk locales default-jdk Prefer: xorg-x11-libs libpng fam mozilla mozilla-nss xorg-x11-Mesa Prefer: unixODBC libsoup glitz java-1_4_2-sun gnome-panel Prefer: desktop-data-SuSE gnome2-SuSE mono-nunit gecko-sharp2 Prefer: apache2-prefork openmotif-libs ghostscript-mini gtk-sharp Prefer: glib-sharp libzypp-zmd-backend mDNSResponder Prefer: -libgcc-mainline -libstdc++-mainline -gcc-mainline-c++ Prefer: -libgcj-mainline -viewperf -compat -compat-openssl097g Prefer: -zmd -OpenOffice_org -pam-laus -libgcc-tree-ssa -busybox-links Prefer: -crossover-office -libgnutls11-dev # alternative pkg-config implementation Prefer: -pkgconf Prefer: -openrc Prefer: -file-rc Conflict: ghostscript-library:ghostscript-mini Ignore: sysvinit:initscripts Ignore: aaa_base:aaa_skel,suse-release,logrotate,ash,mingetty,distribution-release Ignore: gettext-devel:libgcj,libstdc++-devel Ignore: pwdutils:openslp Ignore: pam-modules:resmgr Ignore: rpm:suse-build-key,build-key Ignore: bind-utils:bind-libs Ignore: alsa:dialog,pciutils Ignore: portmap:syslogd Ignore: fontconfig:freetype2 Ignore: fontconfig-devel:freetype2-devel Ignore: xorg-x11-libs:freetype2 Ignore: xorg-x11:x11-tools,resmgr,xkeyboard-config,xorg-x11-Mesa,libusb,freetype2,libjpeg,libpng Ignore: apache2:logrotate Ignore: arts:alsa,audiofile,resmgr,libogg,libvorbis Ignore: kdelibs3:alsa,arts,pcre,OpenEXR,aspell,cups-libs,mDNSResponder,krb5,libjasper Ignore: kdelibs3-devel:libvorbis-devel Ignore: kdebase3:kdebase3-ksysguardd,OpenEXR,dbus-1,dbus-1-qt,hal,powersave,openslp,libusb Ignore: kdebase3-SuSE:release-notes Ignore: jack:alsa,libsndfile Ignore: libxml2-devel:readline-devel Ignore: gnome-vfs2:gnome-mime-data,desktop-file-utils,cdparanoia,dbus-1,dbus-1-glib,krb5,hal,libsmbclient,fam,file_alteration Ignore: libgda:file_alteration Ignore: gnutls:lzo,libopencdk Ignore: gnutls-devel:lzo-devel,libopencdk-devel Ignore: pango:cairo,glitz,libpixman,libpng Ignore: pango-devel:cairo-devel Ignore: cairo-devel:libpixman-devel Ignore: libgnomeprint:libgnomecups Ignore: libgnomeprintui:libgnomecups Ignore: orbit2:libidl Ignore: orbit2-devel:libidl,libidl-devel,indent Ignore: qt3:libmng Ignore: qt-sql:qt_database_plugin Ignore: gtk2:libpng,libtiff Ignore: libgnomecanvas-devel:glib-devel Ignore: libgnomeui:gnome-icon-theme,shared-mime-info Ignore: scrollkeeper:docbook_4,sgml-skel Ignore: gnome-desktop:libgnomesu,startup-notification Ignore: python-devel:python-tk Ignore: gnome-pilot:gnome-panel Ignore: gnome-panel:control-center2 Ignore: gnome-menus:kdebase3 Ignore: gnome-main-menu:rug Ignore: libbonoboui:gnome-desktop Ignore: postfix:pcre Ignore: docbook_4:iso_ent,sgml-skel,xmlcharent Ignore: control-center2:nautilus,evolution-data-server,gnome-menus,gstreamer-plugins,gstreamer,metacity,mozilla-nspr,mozilla,libxklavier,gnome-desktop,startup-notification Ignore: docbook-xsl-stylesheets:xmlcharent Ignore: liby2util-devel:libstdc++-devel,openssl-devel Ignore: yast2:yast2-ncurses,yast2-theme-SuSELinux,perl-Config-Crontab,yast2-xml,SuSEfirewall2 Ignore: yast2-core:netcat,hwinfo,wireless-tools,sysfsutils Ignore: yast2-core-devel:libxcrypt-devel,hwinfo-devel,blocxx-devel,sysfsutils,libstdc++-devel Ignore: yast2-packagemanager-devel:rpm-devel,curl-devel,openssl-devel Ignore: yast2-devtools:perl-XML-Writer,libxslt,pkgconfig Ignore: yast2-installation:yast2-update,yast2-mouse,yast2-country,yast2-bootloader,yast2-packager,yast2-network,yast2-online-update,yast2-users,release-notes,autoyast2-installation Ignore: yast2-bootloader:bootloader-theme Ignore: yast2-packager:yast2-x11 Ignore: yast2-x11:sax2-libsax-perl Ignore: openslp-devel:openssl-devel Ignore: java-1_4_2-sun:xorg-x11-libs Ignore: java-1_4_2-sun-devel:xorg-x11-libs Ignore: kernel-um:xorg-x11-libs Ignore: tetex:xorg-x11-libs,expat,fontconfig,freetype2,libjpeg,libpng,ghostscript-x11,xaw3d,gd,dialog,ed Ignore: yast2-country:yast2-trans-stats Ignore: susehelp:susehelp_lang,suse_help_viewer Ignore: mailx:smtp_daemon Ignore: cron:smtp_daemon Ignore: hotplug:syslog Ignore: pcmcia:syslog Ignore: avalon-logkit:servlet Ignore: jython:servlet Ignore: ispell:ispell_dictionary,ispell_english_dictionary Ignore: aspell:aspel_dictionary,aspell_dictionary Ignore: smartlink-softmodem:kernel,kernel-nongpl Ignore: OpenOffice_org-de:myspell-german-dictionary Ignore: mediawiki:php-session,php-gettext,php-zlib,php-mysql,mod_php_any Ignore: squirrelmail:mod_php_any,php-session,php-gettext,php-iconv,php-mbstring,php-openssl Ignore: simias:mono(log4net) Ignore: zmd:mono(log4net) Ignore: horde:mod_php_any,php-gettext,php-mcrypt,php-imap,php-pear-log,php-pear,php-session,php Ignore: xerces-j2:xml-commons-apis,xml-commons-resolver Ignore: xdg-menu:desktop-data Ignore: nessus-libraries:nessus-core Ignore: evolution:yelp Ignore: mono-tools:mono(gconf-sharp),mono(glade-sharp),mono(gnome-sharp),mono(gtkhtml-sharp),mono(atk-sharp),mono(gdk-sharp),mono(glib-sharp),mono(gtk-sharp),mono(pango-sharp) Ignore: gecko-sharp2:mono(glib-sharp),mono(gtk-sharp) Ignore: vcdimager:libcdio.so.6,libcdio.so.6(CDIO_6),libiso9660.so.4,libiso9660.so.4(ISO9660_4) Ignore: libcdio:libcddb.so.2 Ignore: gnome-libs:libgnomeui Ignore: nautilus:gnome-themes Ignore: gnome-panel:gnome-themes Ignore: gnome-panel:tomboy Substitute: utempter %ifnarch s390 s390x ppc ia64 Substitute: java2-devel-packages java-1_4_2-sun-devel %else %ifnarch s390x Substitute: java2-devel-packages java-1_4_2-ibm-devel %else Substitute: java2-devel-packages java-1_4_2-ibm-devel xorg-x11-libs-32bit %endif %endif Substitute: yast2-devel-packages docbook-xsl-stylesheets doxygen libxslt perl-XML-Writer popt-devel sgml-skel update-desktop-files yast2 yast2-devtools yast2-packagemanager-devel yast2-perl-bindings yast2-testsuite # # SUSE compat mappings # Substitute: gcc-c++ gcc Substitute: libsigc++2-devel libsigc++-2.0-dev Substitute: glibc-devel-32bit Substitute: pkgconfig pkg-config %ifarch %ix86 Substitute: kernel-binary-packages kernel-default kernel-smp kernel-bigsmp kernel-debug kernel-um kernel-xen kernel-kdump %endif %ifarch ia64 Substitute: kernel-binary-packages kernel-default kernel-debug %endif %ifarch x86_64 Substitute: kernel-binary-packages kernel-default kernel-smp kernel-xen kernel-kdump %endif %ifarch ppc Substitute: kernel-binary-packages kernel-default kernel-kdump kernel-ppc64 kernel-iseries64 %endif %ifarch ppc64 Substitute: kernel-binary-packages kernel-ppc64 kernel-iseries64 %endif %ifarch s390 Substitute: kernel-binary-packages kernel-s390 %endif %ifarch s390x Substitute: kernel-binary-packages kernel-default %endif %define debian_version 800 Macros: %debian_version 800
Visit webUI to check project configuration
Create an OBS project linked to DoD
$ osc -A https://stretch:443 meta prj test -e
<project name=”test”>
<title>test</title>
<description>test</description>
<person userid=”Admin” role=”maintainer”/>
<repository name=”Debian_8.0″>
<path project=”Debian:8″ repository=”main”/>
<arch>x86_64</arch>
</repository>
</project>
Visit webUI to check project configuration
Adding a package to the project
$ osc -A https://stretch:443 co test ; cd test $ mkdir hello ; cd hello ; apt-get source -d hello ; cd - ; $ osc add hello $ osc ci -m "New import" hello
The package should go to dispatched state then get in blocked state while it downloads build dependencies from DoD link, eventually it should start building. Please check the journal logs to check if something went wrong or gets stuck.
Visit webUI to check hello package build state
OBS logging to the journal
Check in the journal logs everything went fine:
$ sudo journalctl -u obsdispatcher.service -u obsdodup.service -u obsscheduler@x86_64.service -u obsworker.service -u obspublisher.service
Troubleshooting
Currently we are facing few issues with web UI:
- Projects are not listed in the general project page (#839137)
- Packages are not listed in the project page
- Statistical Plots are not being drawn in monitor page nor system status in main page
- Text editor does not save modified files (#839133)
- Click on patchinfo/delete in a project page gets 404 (#839143)
- Search function does not work (#839137)
And there are more issues that have not been reported, please do ‘reportbug obs-api‘.
