×

Notice

The forum is in read only mode.
  1. Forum
  3. Development
  5. Development
  7. Docker containers for building deviation

Docker containers for building deviation

More
23 Apr 2016 10:41 #47033 by mwm
Replied by mwm on topic Docker containers for building deviation
It sounds like you are targeting these at end users, not developers. Which means I won't be using them. Even if we get a Linux image running on my desktop, running emulator builds in a VM sucks, and I have no intention of going back to that.

Being at the end of a slow DSL line makes downloading docker images painful enough that doing it just to check out something I don't expect to use isn't a priority. I'll do it at some point after things have settled down.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.
More
23 Apr 2016 12:31 #47041 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I am actually targeting these images at everyone. I plan to stop supporting any other method of building Deviation in the future. The build environment is finicky and unless the 7e/F7 goes completely to the wayside, we'll never completely get rid of the compiler-size mattering (because we're still stuck with protocol modules). If we do upgrade to a different compiler, than I want everyone to do it at the same time.
I've done the build on Windows now (using the VM) and it is quite smooth, and seems no slower than when I built though MingW (though that is subjective).

So if developers or users don't want to use the recommended build environment, that is ok, but I expect to shutdown all build issues with 'you didn't use the docker image'.

Note that I'm not there yet. This is still in a pioneering stage right now.
More
25 Apr 2016 18:16 #47175 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Hi, changed my HDD and re-installed windows, so have a working Docker installation.

The "standard" build (with no local git version) builds correctly, however, with my "local" repository, I have the following errors :
Building zip_devo7e zip_win_emu_devo7e
Preparing for ARM build
Preparing for Windows build
/usr/bin/env: perl : No such file or directory
/usr/bin/env: python : No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2

What would be the best way to correct that ? install perl and python on the PC ? (I don't think so), or correct in some way the container ?
More
25 Apr 2016 19:09 #47180 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
how did you create the ocker image (what was your 'docker run' command)?
if you like, you can attach the result of 'docker inspect deviation_build' (but please look at the result to make sure there is no personal info in there)
More
25 Apr 2016 19:35 #47182 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
I'm on Windows, command line was "docker run -it -v ~/Documents:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker"

Last lines were (starting from a "fresh" container
/usr/bin/install -c -d /root/portaudio-w32/lib/pkgconfig
/usr/bin/install -c -m 644 portaudio-2.0.pc /root/portaudio-w32/lib/pkgconfig/portaudio-2.0.pc
------------------------------------------------------------
PortAudio was successfully installed.
On some systems (e.g. Linux) you should run 'ldconfig' now
to make the shared object available.  You may also need to
modify your LD_LIBRARY_PATH environment variable to include
the directory /root/portaudio-w32/lib
------------------------------------------------------------
make install-recursive
make[1]: Entering directory `/root/src/portaudio'
if test -n "" ; then for dir in ""; do make -C $dir install; done ; fi
make[1]: Leaving directory `/root/src/portaudio'
/usr/bin/env: perl : No such file or directory
/usr/bin/env: python : No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2


As for the inspection, what values are considered personal ? (There are a few hashes or IDs, which I don't know if they are...)
More
25 Apr 2016 19:57 #47183 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
the hashes aren't personal. anything with your real name or if you don't want your computer userid exposed would be things to look out for.

The errors you see don't make much sense. I wonder if this is a cr/lf issue (the scripts have a '^M' when you downloaded onto windows and in unix that messes up the execution). can you press the 'shell' button and try to execute:
/usr/bin/env perl
and
/usr/bin/env python

and make sure both give you an interactive prompt?
(you can usually use CTRL-D to exit the interactive prompt you get)

I'll try to test it on Windows when I get home. I hadn't tested using a mounted get repo previously
More
25 Apr 2016 20:07 - 25 Apr 2016 20:19 #47184 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
In the shell, both work wonderfully and give interactive prompts.

However, once the "build" has started and failed, while perl command keeps working, the python one is KO
Last edit: 25 Apr 2016 20:19 by TheSFReader.
More
26 Apr 2016 11:56 #47229 by mwm
Replied by mwm on topic Docker containers for building deviation
If /usr/bin/env isn't finding perl or python in one environment, but it's working in another, I'd suspect the PATH is broken in the one that isn't working.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.

My remotely piloted vehicle ("drone") is a yacht.
More
26 Apr 2016 14:03 #47239 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I'm pretty sure it is simply that when you use git on windows, it adds CRLF characters to the end of the lines. Bash will not execute a script with CRLF endings. fixing it in the image is non-trivial. It may be easiest to just require developers to set the core.autocrlf setting to false. Also, there appears to be .gitattributes which may allow crlf configuration on a per file basis. I'll play with it and see if I can figure out a solution.
More
27 Apr 2016 01:58 #47281 by hexfet
Replied by hexfet on topic Docker containers for building deviation
I'm using the docker image with ubuntu 16.04. Had to add the libfltk1.3-dev package to build emulators in the shell.

Is it possible to make the build and shell run as the user that started the container instead of root?
More
27 Apr 2016 03:40 #47285 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
Hmm...it isn't straight forward, because the container has no knowledge of the outer world. But the mounted filesystem preserves its attributes (which includes the directory owner uid). So we know who owned the directory initially, and could either try to create a user with the same uid. Maybe docker offers a better solution, I'll look into it.

I figured linux emulators wouldn't be too common so I didn't include them, but I could add support to auto-setup the build env quite eaisly.
More
27 Apr 2016 19:23 - 27 Apr 2016 19:35 #47326 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Hi,
Just to let you know, I've tried and "Dos2Unix" all files before launching a build and as PB thought, it built correctly. So now it leaves us to find which ones need to be either fixed, or specifically stored/used with "unix" line ending forced.

(Beginning to converge, utils and src/libopencm3 directories for now) (but not sufficient)
Last edit: 27 Apr 2016 19:35 by TheSFReader.
More
27 Apr 2016 20:31 #47329 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
it should only be the .py and .pl files in /utils.

I just added a .gitattributes file to the repo. can you pull the latest changes (in windows) and see if that fixes the issue in the image?
More
27 Apr 2016 20:57 #47331 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Started "fresh" with the updated files --> failed. (but I didn't start from a fresh git fork/clone. Should I ?
More
27 Apr 2016 22:17 #47335 by HappyHarry
Replied by HappyHarry on topic Docker containers for building deviation
i just tested building a container using a local git clone and it fails when building portaudio, well it builds it ok but something falls over when it's doing install-recursive, building a normal container works perfectly, and building a normal container and a container with a local git clone works fine on linux. here's the error
------------------------------------------------------------
PortAudio was successfully installed.

On some systems (e.g. Linux) you should run 'ldconfig' now
to make the shared object available.  You may also need to
modify your LD_LIBRARY_PATH environment variable to include
the directory /root/portaudio-w32/lib
------------------------------------------------------------

make install-recursive
make[1]: Entering directory `/root/src/portaudio'
if test -n "" ; then for dir in ""; do make -C $dir install; done ; fi
make[1]: Leaving directory `/root/src/portaudio'
: No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2

phil@Anubis-Desk MINGW64 ~

any idea what's up?
More
27 Apr 2016 23:28 #47337 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
For you windows folks, try this:
In windows (NOT in the Docker image!)
go into your git/deviation driectory and remove everything but the .git directory
run:
git pull
git checkout

now try to build in docker and see how it goes.
to verify that things are wokring, you should be able to drop to a shell in the docker image and do:
file /git/deviation/src/libopencm3/scripts/irq2nvic_h
/git/deviation/src/libopencm3/scripts/irq2nvic_h: Python script, ASCII text executable
file /git/deviation/utils/dfu.py
/git/deviation/utils/dfu.py: Python script, ASCII text executable
file /git/deviation/README
/git/deviation/README: ASCII text, with CRLF line terminators
Note that the 1st 2 should not have 'CRLF' and the README file should (this is only for Windows docker users. For users running docker on linux none of the files will have CRLF.
More
28 Apr 2016 04:49 #47345 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
Unfortunately, I had to create a new docker image, as the last set of changes could not be done only in the python script. I don't think the 0.9.3 python script will be compatible with the old image, so if you see errors, it is just another indication that you need to update the image:
docker rm deviation_build
docker pull deviationtx/deviation-docker
Then follow the instructions for 'docker run ...' here:
www.deviationtx.com/forum/7-development/...ding-deviation#46833
or
www.deviationtx.com/forum/7-development/...ding-deviation#46759

This should be the last time I need to update the docker image, I hope.

The big change here is that the build process will no longer run as 'root' but instead as a user called 'docker' This user should have the same userid/groupid as the user who owns the git repository. This should make things work much more smoothly when using a mounted git repo on Linux/Mac (though Mac is still untested). I did a bunch of experimentation with Windows builds too, and they all work for me now, so I'd like any feedback on that as well. I also included the gcc cross compiler in the image since it seems to take a long time to download for me, and I was sic of constantly re-fetching it. I left the windows libs as fetch on demand for now since they don't take as long, and many users may never need to build them.
More
28 Apr 2016 07:20 #47349 by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
This time it's OK :)

Thanks :)
More
28 Apr 2016 17:02 #47374 by hexfet
Replied by hexfet on topic Docker containers for building deviation
Just installed the new image. Getting the errors below on Ubuntu with both the build button and running make from the command line. The git and releases trees show up as owned by docker:docker and most file permissions are 775. Maybe I have something non-standard somewhere.

uilding zip_devo10 zip_devo12 zip_devo6 zip_devo7e zip_devo8 zip_devof12e zip_devof4 zip_devof7
Preparing for ARM build
GENHDR include/libopencm3/lpc17xx/irq.json
Traceback (most recent call last):
File "./scripts/irq2nvic_h", line 174, in <module>
main()
File "./scripts/irq2nvic_h", line 171, in main
convert(open(infile), open(nvic_h, 'w'), open(vector_nvic_c, 'w'), open(cmsis, 'w'))
IOError: [Errno 13] Permission denied: './include/libopencmsis/lpc17xx/irqhandlers.h'
make[2]: *** [include/libopencm3/lpc17xx/irq.json.genhdr] Error 1
make[1]: *** [libopencm3/lib/libopencm3_stm32f1.a] Error 2
make: *** [zip_devo10] Error 2
More
28 Apr 2016 17:33 - 28 Apr 2016 17:34 #47376 by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
drop to a shell inside the docker image, and do:
ls -l /git/deviation/src/libopencm3/scripts/irq2nvic_h
ls -l  /git/deviation/src/libopencm3/include/libopencmsis/lpc17xx/irqhandlers.h
ls -lad  /git/deviation/src/libopencm3/include/libopencmsis/lpc17xx

file  /git/deviation/src/libopencm3/scripts/irq2nvic_h
I am betting that some file/directory has corrupted permission (possibly from using docker previously on your tree). you could also just try the following form inside docker (or outside docker using the proper path). Be REALLY careful with that rm -rf
sudo rm -rf /git/deviation/*;cd /git/deviation; git checkout master
Last edit: 28 Apr 2016 17:34 by PhracturedBlue.
  1. Forum
  3. Development
  5. Development
  7. Docker containers for building deviation
Time to create page: 0.201 seconds
Powered by Kunena Forum
Force desktop display