Docker containers for building deviation

More
21 Mar 2016 08:36 #44906 by mwm
I've just pushed a set of docker containers for building deviation to hub.docker.com/u/mwmeyer/ . If you're not familiar with it, docker is a lightweight container system designed for running commands - and not systems - in virtual machines. It's perfectly reasonable - and even normal - to spin up a docker container to to do part of a build process, and then remove it when that's done. They don't have the overhead of running an operating system, and expect to run in the CLI, so they are easy to integrate into an existing development environment, as opposed to having to deal with a second machine.

See the description of mwmeyer/deviation description for some suggested uses for these. But the short version is that you'll want to run containers that mount your local source tree as a volume and build against that, as they don't include sources. This means the images never change.

There are four of them:
  1. mwmeyer/deviation - has the tools recommended in the deviation README to build transmitter images and zips of the same.
  2. mwmeyer/deviation-emulators - this adds the tools needed to build the windows emulators and zip files. The command "make release" in this container will build everything you'd want to upload.
  3. mwmeyer/deviation-manual - This only has tools for building the manuals.
  4. mwmeyer/deviation-sources - This is a data-only image that comes pre-populated with the sources for deviation and the manual. You probably don't want this unless you're going to be building in the cloud.

This is a different type of solution than the existing virtualbox image. Which you'd prefer will depend a lot on you and your preferred tools.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.

Please Log in or Create an account to join the conversation.

More
21 Mar 2016 12:02 #44919 by Antfurn
Replied by Antfurn on topic Docker containers for building deviation
Awesome, thanks for creating these.

Please Log in or Create an account to join the conversation.

More
25 Mar 2016 04:35 #45140 by silpstream
Replied by silpstream on topic Docker containers for building deviation
This is great! Any chance you can share a dockerfile for this or the steps/commits you made to create it? Also, can this be modified to build the modified bootloader you are working on?

Thanks very much!

Please Log in or Create an account to join the conversation.

More
25 Mar 2016 19:42 #45180 by mwm
Yeah, I've been meaning to share those. Just haven't done it yet. I've attached a zip file with everything you need to create them except gcc-arm-none-eabi-4_8-2013q4-20131204-linux.tar.bz2, which you'll want to download from launchpad.net.

The one odd thing is the "deviate" script, which was designed as a simple "build a branch" command. You can safely ignore it if yu want.

As for the dfuse tool, they can be used to build those. You'll want to mount the sources on the container, and you'll have to make sure the libopencm3 sources as well as the bootloader are in it. The toolchain is the same one, though.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.
Attachments:

Please Log in or Create an account to join the conversation.

More
25 Mar 2016 20:16 #45186 by boblethai
Replied by boblethai on topic Docker containers for building deviation
Hi,

I'm quite new in docker (and deviation as well, I must admit!)

The given command does not work for me:

[root@budapest Deviation_Build_Docker]# docker run deviation -v ~/src/deviation:/deviation --name deviation mwmeyer/deviation /bin/bash
Unable to find image 'deviation:latest' locally
Trying to pull repository docker.io/library/deviation ... not found
Error: image library/deviation:latest not found

but if I do:

[root@budapest ~]# docker run --rm -i -t -v ~/src/deviation:/mnt mwmeyer/deviation /bin/bash
root@e4eada67c324:/deviation/src#

it looks better.

My docker version is:
[root@budapest ~]# docker --version
Docker version 1.8.2-el7.centos, build a01dc02/1.8.2

probably not the most up to date... because I'm running centos 7.

Bob

Please Log in or Create an account to join the conversation.

More
25 Mar 2016 21:41 #45191 by mwm
Well, this is my first docker project, so I'm not in much better shape than you are. For the record, I'm running it on a Mac.

However, your first command line should start "docker run -v ...", not "docker run deviation -v ...". That extra "deviation" becomes the image name, which makes it search for it in the docker.io/library instead of my account, hence the error message.

I've fixed the description to be correct (which means add "-it" as well). Thanks for reporting it.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.

Please Log in or Create an account to join the conversation.

More
26 Mar 2016 14:36 #45222 by silpstream
Replied by silpstream on topic Docker containers for building deviation
Thanks so much for the files mwm!

I used Kitematic to link the volume and start up the container and then went to the cli ("docker cli" button on the bottom left of Kitematic) and did a "docker attach deviation" to get inside.

I then ran "make zip TARGET=devoXX" (replace XX with model number) from the src folder. This created a zip file with everything I needed to flash using the 0.8 version of uploader tool. Remember to run "make distclean" after to clean things up for the next build.

Please Log in or Create an account to join the conversation.

More
18 Apr 2016 21:09 - 19 Apr 2016 03:04 #46759 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I just built a new docker image here:
hub.docker.com/r/deviationtx/deviation-docker/

The docker-file and build script can be found here:
github.com/DeviationTX/deviation-docker

1st pull the image:
docker pull deviationtx/deviation-docker

next make a directory where you want the builds to go on your host machine:
mkdir ~/devo_builds

now you can start the image in a couple ways:
As a developer where you already have a cloned deviation repo
docker run -it -v <path to git repo head>:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker
NOTE: <git repo head> is the directory above your repo (it should contain a subdirectory called 'deviation')

As a user without a cloned repo:
docker run -it -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker

On subsequent runs, you can do:
docker start -i deviation_build

The scripts will 1st check whether there have been any updates to the build script recently. It will then update your deviation repo (or clone it if you didn't provide one). next it will show a menu that will let you choose what you want to build. Third it will fetch any additional files needed to run the build. Lastly it will build the selected targets and store the results in the ~/devo_builds dir on your host machine.

The build env will be cached, so the next time you run, it will be much faster.

The build script is still very crude. I will provide options to further configure the builds in the future. I also need to test building on Windows, and write some instructions, but my understanding is that I should be able to build a lightweight installer to automate all of that.

If this works, I plan to make it the only supported build system. This image should be nearly identical to the version used by our official and nightly builds, so there should be no cases where a build works in one place and not in another.

The primary downside is that the file image is quite large.

I'd like any feedback if you are successful with this method.
Last edit: 19 Apr 2016 03:04 by PhracturedBlue.

Please Log in or Create an account to join the conversation.

More
19 Apr 2016 02:39 - 19 Apr 2016 02:50 #46774 by silpstream
Replied by silpstream on topic Docker containers for building deviation

PhracturedBlue wrote: now you can start the image in a couple ways:
As a developer where you already have a cloned deviation repo
docker run -it -v <path to git repo head>:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker
NOTE: <git repo head> is the directory above your repo (it should contain a subdirectory called 'deviation')

As a user without a cloned repo:
docker run -it -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker

On subsequent runs, you can do:
docker start -i deviation

Very nice. Having the interface will go a long way to easing the build process. Especially if you are going to start adding in build time options later on.

Worked for me on "devo7e". "devo7e-256" throws up an error "No rule to make target `zip_devo7e-256'".

You have a typo error for subsequent runs above, should read as "docker start -i deviation_build", if not mwm's docker container gets started instead.

I've not looked into the scripts you have.

I'm wondering, for those of us developing on forked clones (most developers would be right?), I assume that it doesn't actually do any updating and will not switch me back to master branch. Developer forks are probably not a use case right?

Thanks for this! :cheer:
Last edit: 19 Apr 2016 02:50 by silpstream.

Please Log in or Create an account to join the conversation.

More
19 Apr 2016 03:10 #46775 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
It should work on developer forks, and yes they are absolutely a use case. Ideally everyone will use these images (once they are fully baked). Note that you MUST use an external git repo and the '-v <...>:/git' option if you use a non-standard repo. It WILL run a 'git pull' 1st. Maybe I'll disable that if any modifications are detected.

The current proff of concept GUI is using 'dialog' which is very limited in its capability. I found something called 'snack' for python which allows building more complicated text-based GUIs. I'm going to rework the current script into a python script which will allow me to add some developer-centric options. I will take a look at the devo7e-256. Not sure why that wouldn't work.

Please Log in or Create an account to join the conversation.

More
19 Apr 2016 13:08 #46794 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
the devo7e-256 thing was a deviation bug and had nothing to do with docker. If you run the image again, it should build now.

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 01:11 #46830 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I updated the docker image today and replaced the shell based gui with a python based one. It doesn't actually work exactly as I want, but it is close, and will add a ot more flexibility for the future. You'll need a new image since I changed the pre-installed packages. Hopefully for the last time.

The issue I have is that I want the 'Update GIT' button to run a pull and return to the menu. This works fine on my linux machine, but for some reason in docker it hangs. I've temporarily changed the code to exit after an update to avoid the hang, which isn't ideal, but it will do for the time being until I figure out what is wrong in docker.

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 04:06 #46833 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
ok, I have tested out docker on Windows, and it seems to work.

Windows steps are nearly identical to linux:

Install docker for windows. You must have a 64-bit windows installation of a supported Windows variant:
Instructions here: docs.docker.com/engine/installation/windows/

If you already have VirtualBox installed, make sure it is at the latest version before you begin AND unselect it from the list during docker install. If you don't have the latest VirtualBox, the install will pass, but initialization will fail. It took me a while to figure out what was going on

now click 'Docker Quickstart Terminal'
When you eventually get a prompt (this is a one-time setup), run:
docker pull deviationtx/deviation-docker

And then:
mkdir /c/Users/<username>/Release
now:
docker run -it --name deviation_build -v /c/Users/<username>/Release:/release deviationtx/deviation-docker

and next time:
docker start -i deviation_build

Now the question is: Is this too much work to be worth it? Is there a better solution, or a way to simplify the process?

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 07:14 #46845 by mwm
Note that the OSX version of docker uses the same technology that Windows does, running containers on a virtualbox image, and dealing with the installation of virtualbox if you need to. The instructions are virtually identical to the Windows ones.

I guess the question is - how hard is this compared to installing a build environment "by hand". For the Mac, I believe it's much easier, but never bothered to do that as we don't have instructions for it. For windows, it seems a bit easier, but again - I haven't done that. For linux, it fixes the issue of making sure you have the right build chain, which has gotten a bit painful to deal with.

Are these 32-bit or 64 bit Linux images? The ones I built are 64 bits and can't work on my BSD desktop.

Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 08:58 - 20 Apr 2016 09:00 #46854 by sbstnp
Replied by sbstnp on topic Docker containers for building deviation
Most of the time for a docker container to access outside world the simplest solution is to use networking type as 'host' - this will make the container inherit the network configuration - same interfaces, same IP settings and routes. This way there's no need to fiddle with SNAT rules, at least on Linux. This can be different on other OSes though.

So, try with docker run [...] --net=host [...], should allow git to fetch changes.

Devo 10 + 4in1
Spektrum Dx9
FrSky Taranis + TBS Crossfire
Last edit: 20 Apr 2016 09:00 by sbstnp.

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 12:33 #46863 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
Even on Window we've had trouble with the build chain not working on occasion, so I really like the idea of only having a single build environment.

I think all docker images are by default 64-bit. I am not sure how to even change it. Why does BSD not support 64bit images?

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 15:11 #46877 by richardclli
Replied by richardclli on topic Docker containers for building deviation
Nice work, docker will help out people that want to build deviation without the need to install the tool chain by themselves!

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 17:53 #46887 by silpstream
Replied by silpstream on topic Docker containers for building deviation
@PhracturedBlue. This is great! I got things working on OS X for devo7e, devo7e-256 and the devo7e emulator. Just 3 things:

1) I cannot update from git, I'm guessing it might be because I'm using my own repository and am using ssh keys, maybe?
Updating build.py
Warning: Permanently added the RSA host key for IP address '192.30.252.128' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
2) Any chance you can give us the ability to get out of the snack gui and into command line within the container. It is useful to be able to operate from command line within the build environment when developing.
3) Could we possibly also specify the git branch prior to pull. Makes life easier when testing code in a branch. Of course if (2) comes true, then we should be okay.

Thanks again! :)

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 18:27 #46889 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I'm out of space on the current gui page. I'll need to think about how to add more options on a 2nd page. In developer mode, I assumed you would manage your git repo yourself, and just use the docker image for building. so just turn off auto-update and setup the repo as you want before you run it. I think switching branches is probably too much to ask of the little gui, but adding a shell option should be reasonably easy to to, so I'll look into that

Please Log in or Create an account to join the conversation.

More
20 Apr 2016 18:33 #46891 by silpstream
Replied by silpstream on topic Docker containers for building deviation
Agreed, just someway to get into shell would be more than enough. Thanks!

Please Log in or Create an account to join the conversation.

Time to create page: 0.134 seconds
Powered by Kunena Forum