Docker containers for building deviation
- mwm
-
Topic Author
- Offline
Less
More
23 Apr 2016 10:41 #47033
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic Docker containers for building deviation
It sounds like you are targeting these at end users, not developers. Which means I won't be using them. Even if we get a Linux image running on my desktop, running emulator builds in a VM sucks, and I have no intention of going back to that.
Being at the end of a slow DSL line makes downloading docker images painful enough that doing it just to check out something I don't expect to use isn't a priority. I'll do it at some point after things have settled down.
Being at the end of a slow DSL line makes downloading docker images painful enough that doing it just to check out something I don't expect to use isn't a priority. I'll do it at some point after things have settled down.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
23 Apr 2016 12:31 #47041
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I am actually targeting these images at everyone. I plan to stop supporting any other method of building Deviation in the future. The build environment is finicky and unless the 7e/F7 goes completely to the wayside, we'll never completely get rid of the compiler-size mattering (because we're still stuck with protocol modules). If we do upgrade to a different compiler, than I want everyone to do it at the same time.
I've done the build on Windows now (using the VM) and it is quite smooth, and seems no slower than when I built though MingW (though that is subjective).
So if developers or users don't want to use the recommended build environment, that is ok, but I expect to shutdown all build issues with 'you didn't use the docker image'.
Note that I'm not there yet. This is still in a pioneering stage right now.
I've done the build on Windows now (using the VM) and it is quite smooth, and seems no slower than when I built though MingW (though that is subjective).
So if developers or users don't want to use the recommended build environment, that is ok, but I expect to shutdown all build issues with 'you didn't use the docker image'.
Note that I'm not there yet. This is still in a pioneering stage right now.
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
25 Apr 2016 18:16 #47175
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Hi, changed my HDD and re-installed windows, so have a working Docker installation.
The "standard" build (with no local git version) builds correctly, however, with my "local" repository, I have the following errors :
What would be the best way to correct that ? install perl and python on the PC ? (I don't think so), or correct in some way the container ?
The "standard" build (with no local git version) builds correctly, however, with my "local" repository, I have the following errors :
Building zip_devo7e zip_win_emu_devo7e
Preparing for ARM build
Preparing for Windows build
/usr/bin/env: perl : No such file or directory
/usr/bin/env: python : No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2What would be the best way to correct that ? install perl and python on the PC ? (I don't think so), or correct in some way the container ?
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
25 Apr 2016 19:09 #47180
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
how did you create the ocker image (what was your 'docker run' command)?
if you like, you can attach the result of 'docker inspect deviation_build' (but please look at the result to make sure there is no personal info in there)
if you like, you can attach the result of 'docker inspect deviation_build' (but please look at the result to make sure there is no personal info in there)
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
25 Apr 2016 19:35 #47182
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
I'm on Windows, command line was "docker run -it -v ~/Documents:/git -v ~/devo_builds:/release --name deviation_build deviationtx/deviation-docker"
Last lines were (starting from a "fresh" container
As for the inspection, what values are considered personal ? (There are a few hashes or IDs, which I don't know if they are...)
Last lines were (starting from a "fresh" container
/usr/bin/install -c -d /root/portaudio-w32/lib/pkgconfig
/usr/bin/install -c -m 644 portaudio-2.0.pc /root/portaudio-w32/lib/pkgconfig/portaudio-2.0.pc
------------------------------------------------------------
PortAudio was successfully installed.
On some systems (e.g. Linux) you should run 'ldconfig' now
to make the shared object available. You may also need to
modify your LD_LIBRARY_PATH environment variable to include
the directory /root/portaudio-w32/lib
------------------------------------------------------------
make install-recursive
make[1]: Entering directory `/root/src/portaudio'
if test -n "" ; then for dir in ""; do make -C $dir install; done ; fi
make[1]: Leaving directory `/root/src/portaudio'
/usr/bin/env: perl : No such file or directory
/usr/bin/env: python : No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2As for the inspection, what values are considered personal ? (There are a few hashes or IDs, which I don't know if they are...)
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
25 Apr 2016 19:57 #47183
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
the hashes aren't personal. anything with your real name or if you don't want your computer userid exposed would be things to look out for.
The errors you see don't make much sense. I wonder if this is a cr/lf issue (the scripts have a '^M' when you downloaded onto windows and in unix that messes up the execution). can you press the 'shell' button and try to execute:
/usr/bin/env perl
and
/usr/bin/env python
and make sure both give you an interactive prompt?
(you can usually use CTRL-D to exit the interactive prompt you get)
I'll try to test it on Windows when I get home. I hadn't tested using a mounted get repo previously
The errors you see don't make much sense. I wonder if this is a cr/lf issue (the scripts have a '^M' when you downloaded onto windows and in unix that messes up the execution). can you press the 'shell' button and try to execute:
/usr/bin/env perl
and
/usr/bin/env python
and make sure both give you an interactive prompt?
(you can usually use CTRL-D to exit the interactive prompt you get)
I'll try to test it on Windows when I get home. I hadn't tested using a mounted get repo previously
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
25 Apr 2016 20:07 - 25 Apr 2016 20:19 #47184
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
In the shell, both work wonderfully and give interactive prompts.
However, once the "build" has started and failed, while perl command keeps working, the python one is KO
However, once the "build" has started and failed, while perl command keeps working, the python one is KO
Last edit: 25 Apr 2016 20:19 by TheSFReader.
Please Log in or Create an account to join the conversation.
- mwm
-
- Offline
26 Apr 2016 11:56 #47229
by mwm
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Replied by mwm on topic Docker containers for building deviation
If /usr/bin/env isn't finding perl or python in one environment, but it's working in another, I'd suspect the PATH is broken in the one that isn't working.
Do not ask me questions via PM. Ask in the forums, where I'll answer if I can.
My remotely piloted vehicle ("drone") is a yacht.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
26 Apr 2016 14:03 #47239
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
I'm pretty sure it is simply that when you use git on windows, it adds CRLF characters to the end of the lines. Bash will not execute a script with CRLF endings. fixing it in the image is non-trivial. It may be easiest to just require developers to set the core.autocrlf setting to false. Also, there appears to be .gitattributes which may allow crlf configuration on a per file basis. I'll play with it and see if I can figure out a solution.
Please Log in or Create an account to join the conversation.
- hexfet
-
- Offline
Less
More
- Posts: 1868
27 Apr 2016 01:58 #47281
by hexfet
Replied by hexfet on topic Docker containers for building deviation
I'm using the docker image with ubuntu 16.04. Had to add the libfltk1.3-dev package to build emulators in the shell.
Is it possible to make the build and shell run as the user that started the container instead of root?
Is it possible to make the build and shell run as the user that started the container instead of root?
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
27 Apr 2016 03:40 #47285
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
Hmm...it isn't straight forward, because the container has no knowledge of the outer world. But the mounted filesystem preserves its attributes (which includes the directory owner uid). So we know who owned the directory initially, and could either try to create a user with the same uid. Maybe docker offers a better solution, I'll look into it.
I figured linux emulators wouldn't be too common so I didn't include them, but I could add support to auto-setup the build env quite eaisly.
I figured linux emulators wouldn't be too common so I didn't include them, but I could add support to auto-setup the build env quite eaisly.
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
27 Apr 2016 19:23 - 27 Apr 2016 19:35 #47326
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Hi,
Just to let you know, I've tried and "Dos2Unix" all files before launching a build and as PB thought, it built correctly. So now it leaves us to find which ones need to be either fixed, or specifically stored/used with "unix" line ending forced.
(Beginning to converge, utils and src/libopencm3 directories for now) (but not sufficient)
Just to let you know, I've tried and "Dos2Unix" all files before launching a build and as PB thought, it built correctly. So now it leaves us to find which ones need to be either fixed, or specifically stored/used with "unix" line ending forced.
(Beginning to converge, utils and src/libopencm3 directories for now) (but not sufficient)
Last edit: 27 Apr 2016 19:35 by TheSFReader.
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
27 Apr 2016 20:31 #47329
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
it should only be the .py and .pl files in /utils.
I just added a .gitattributes file to the repo. can you pull the latest changes (in windows) and see if that fixes the issue in the image?
I just added a .gitattributes file to the repo. can you pull the latest changes (in windows) and see if that fixes the issue in the image?
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
27 Apr 2016 20:57 #47331
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
Started "fresh" with the updated files --> failed. (but I didn't start from a fresh git fork/clone. Should I ?
Please Log in or Create an account to join the conversation.
- HappyHarry
-
- Offline
Less
More
- Posts: 1136
27 Apr 2016 22:17 #47335
by HappyHarry
Replied by HappyHarry on topic Docker containers for building deviation
i just tested building a container using a local git clone and it fails when building portaudio, well it builds it ok but something falls over when it's doing install-recursive, building a normal container works perfectly, and building a normal container and a container with a local git clone works fine on linux. here's the error
any idea what's up?
------------------------------------------------------------
PortAudio was successfully installed.
On some systems (e.g. Linux) you should run 'ldconfig' now
to make the shared object available. You may also need to
modify your LD_LIBRARY_PATH environment variable to include
the directory /root/portaudio-w32/lib
------------------------------------------------------------
make install-recursive
make[1]: Entering directory `/root/src/portaudio'
if test -n "" ; then for dir in ""; do make -C $dir install; done ; fi
make[1]: Leaving directory `/root/src/portaudio'
: No such file or directory
make[1]: *** [include/libopencm3/efm32/efm32g/irq.json.cleanhdr] Error 127
make: *** [distclean] Error 2
phil@Anubis-Desk MINGW64 ~any idea what's up?
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
27 Apr 2016 23:28 #47337
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
For you windows folks, try this:
In windows (NOT in the Docker image!)
go into your git/deviation driectory and remove everything but the .git directory
run:
git pull
git checkout
now try to build in docker and see how it goes.
to verify that things are wokring, you should be able to drop to a shell in the docker image and do:Note that the 1st 2 should not have 'CRLF' and the README file should (this is only for Windows docker users. For users running docker on linux none of the files will have CRLF.
In windows (NOT in the Docker image!)
go into your git/deviation driectory and remove everything but the .git directory
run:
git pull
git checkout
now try to build in docker and see how it goes.
to verify that things are wokring, you should be able to drop to a shell in the docker image and do:
file /git/deviation/src/libopencm3/scripts/irq2nvic_h
/git/deviation/src/libopencm3/scripts/irq2nvic_h: Python script, ASCII text executable
file /git/deviation/utils/dfu.py
/git/deviation/utils/dfu.py: Python script, ASCII text executable
file /git/deviation/README
/git/deviation/README: ASCII text, with CRLF line terminatorsPlease Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
28 Apr 2016 04:49 #47345
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
Unfortunately, I had to create a new docker image, as the last set of changes could not be done only in the python script. I don't think the 0.9.3 python script will be compatible with the old image, so if you see errors, it is just another indication that you need to update the image:
docker rm deviation_build
docker pull deviationtx/deviation-docker
Then follow the instructions for 'docker run ...' here:
www.deviationtx.com/forum/7-development/...ding-deviation#46833
or
www.deviationtx.com/forum/7-development/...ding-deviation#46759
This should be the last time I need to update the docker image, I hope.
The big change here is that the build process will no longer run as 'root' but instead as a user called 'docker' This user should have the same userid/groupid as the user who owns the git repository. This should make things work much more smoothly when using a mounted git repo on Linux/Mac (though Mac is still untested). I did a bunch of experimentation with Windows builds too, and they all work for me now, so I'd like any feedback on that as well. I also included the gcc cross compiler in the image since it seems to take a long time to download for me, and I was sic of constantly re-fetching it. I left the windows libs as fetch on demand for now since they don't take as long, and many users may never need to build them.
docker rm deviation_build
docker pull deviationtx/deviation-docker
Then follow the instructions for 'docker run ...' here:
www.deviationtx.com/forum/7-development/...ding-deviation#46833
or
www.deviationtx.com/forum/7-development/...ding-deviation#46759
This should be the last time I need to update the docker image, I hope.
The big change here is that the build process will no longer run as 'root' but instead as a user called 'docker' This user should have the same userid/groupid as the user who owns the git repository. This should make things work much more smoothly when using a mounted git repo on Linux/Mac (though Mac is still untested). I did a bunch of experimentation with Windows builds too, and they all work for me now, so I'd like any feedback on that as well. I also included the gcc cross compiler in the image since it seems to take a long time to download for me, and I was sic of constantly re-fetching it. I left the windows libs as fetch on demand for now since they don't take as long, and many users may never need to build them.
Please Log in or Create an account to join the conversation.
- TheSFReader
-
- Offline
Less
More
- Posts: 64
28 Apr 2016 07:20 #47349
by TheSFReader
Replied by TheSFReader on topic Docker containers for building deviation
This time it's OK 
Thanks
Thanks
Please Log in or Create an account to join the conversation.
- hexfet
-
- Offline
Less
More
- Posts: 1868
28 Apr 2016 17:02 #47374
by hexfet
Replied by hexfet on topic Docker containers for building deviation
Just installed the new image. Getting the errors below on Ubuntu with both the build button and running make from the command line. The git and releases trees show up as owned by docker:docker and most file permissions are 775. Maybe I have something non-standard somewhere.
uilding zip_devo10 zip_devo12 zip_devo6 zip_devo7e zip_devo8 zip_devof12e zip_devof4 zip_devof7
Preparing for ARM build
GENHDR include/libopencm3/lpc17xx/irq.json
Traceback (most recent call last):
File "./scripts/irq2nvic_h", line 174, in <module>
main()
File "./scripts/irq2nvic_h", line 171, in main
convert(open(infile), open(nvic_h, 'w'), open(vector_nvic_c, 'w'), open(cmsis, 'w'))
IOError: [Errno 13] Permission denied: './include/libopencmsis/lpc17xx/irqhandlers.h'
make[2]: *** [include/libopencm3/lpc17xx/irq.json.genhdr] Error 1
make[1]: *** [libopencm3/lib/libopencm3_stm32f1.a] Error 2
make: *** [zip_devo10] Error 2
uilding zip_devo10 zip_devo12 zip_devo6 zip_devo7e zip_devo8 zip_devof12e zip_devof4 zip_devof7
Preparing for ARM build
GENHDR include/libopencm3/lpc17xx/irq.json
Traceback (most recent call last):
File "./scripts/irq2nvic_h", line 174, in <module>
main()
File "./scripts/irq2nvic_h", line 171, in main
convert(open(infile), open(nvic_h, 'w'), open(vector_nvic_c, 'w'), open(cmsis, 'w'))
IOError: [Errno 13] Permission denied: './include/libopencmsis/lpc17xx/irqhandlers.h'
make[2]: *** [include/libopencm3/lpc17xx/irq.json.genhdr] Error 1
make[1]: *** [libopencm3/lib/libopencm3_stm32f1.a] Error 2
make: *** [zip_devo10] Error 2
Please Log in or Create an account to join the conversation.
- PhracturedBlue
-
- Offline
Less
More
- Posts: 4402
28 Apr 2016 17:33 - 28 Apr 2016 17:34 #47376
by PhracturedBlue
Replied by PhracturedBlue on topic Docker containers for building deviation
drop to a shell inside the docker image, and do:I am betting that some file/directory has corrupted permission (possibly from using docker previously on your tree). you could also just try the following form inside docker (or outside docker using the proper path). Be REALLY careful with that rm -rf
ls -l /git/deviation/src/libopencm3/scripts/irq2nvic_h
ls -l /git/deviation/src/libopencm3/include/libopencmsis/lpc17xx/irqhandlers.h
ls -lad /git/deviation/src/libopencm3/include/libopencmsis/lpc17xx
file /git/deviation/src/libopencm3/scripts/irq2nvic_hsudo rm -rf /git/deviation/*;cd /git/deviation; git checkout master
Last edit: 28 Apr 2016 17:34 by PhracturedBlue.
Please Log in or Create an account to join the conversation.
Time to create page: 0.095 seconds
-
Home
-
Forum
-
Development
-
Development
- Docker containers for building deviation