Moving build system to pixi#2037
Conversation
A devcontainer is a small wrapper around Docker to make it easy to set up a clean development environment. Most IDE's and code editors support these now as well as many cloud environments. I've included a very simple environment to start with which we can extend as necessary later to streamline the build process.
This diff adds the pixi file and updates some SuperBuild scripts to compile. With this we can run `pixi install` and `pixi run build` and the compilation works.
Output report from before and after pixi are almost identical.
roughly a 6 minute build now
PoissonRecon builds from a hand-written Makefile via `make`, which the minimal pixi container image does not provide (unlike the CI runners, which have a system make on PATH). Without it the Docker build failed at the poissonrecon step with 'make: command not found'.
|
Related: |
This is a clunky solution but give we will want to upstream all of these patches anyway it seems ok for now. It means we can keep them as .patch files making that transition easier. In the future, we probably want to relook at how we handle these external cmake projects but for now I'm avoiding changing too much.
|
The patches were trying to apply every time which as they weren't idempotent meant the build failed. Instead I added a small cmake wrapper which checks if the patch was already applied so repeated builds should now work. |
|
Before this PR, I had never heard of I'm using This article gives a good breakdown of the evolution of Python package management, and how to choose between the latest tools Pretty much: Overall, seems like |
Shapely 2 no longer has `__getitem__` for MultiPolygon, so unary_union results were written whole instead of picking the largest polygon. Commit 75a1e1f only fixed one half of that mismatch by changing the schema to MultiPolygon; Fiona 1.9+ still rejects Polygon records against a MultiPolygon schema. Use .geoms to select the largest polygon and declare the GPKG layer as Polygon again.
|
Instructions for installing pixi on mac use curl, but that fails due to some issue with default ssl / curl / mac os x (I think -- will dig in later). So, I installed pixi using Homebrew, and all was pretty smooth until what I think is a memory issue: |
|
So I ran the full OATS suite overnight just on my dev machine (no GPU, Linux) and the results were fairly promising at least on a pass/fail basis - 48 tests, 11 failures, (attached pic of results to preserve colours). Summarising the failures, there were 3 real bugs. 
Cause A - the Cause B - codem alignment produces no Cause C - So in summary, we have at least 2 outstanding bugs before we even look at the output files although neither of these are exclusive to the pixi change. And the latest image is broken for DEM generation. |
@smathermather So next step would be to find:
cd /Users/user01/Documents/git/ODM
pixi shell
find .pixi -name "libboost_iostreams*.dylib"
|
It seems like we weren't setting the RPATH on MacOS correctly. This change should bake it into the binary matching how it works on Linux.
Yea this was it. I don't have a way to test on Mac easily so I was relying on the smoke test which basically just does As a fix, the MacOS binaries should now include the RPATH similar to how Linux does it. Please let me know if this helped. |
Windows uses a fundamentally different kind of dynamic library loading so we can't easily check it in smoke.py. Instead lets drop it for now.
Further along: |
Declare the standard OCI labels (revision, source, version, created) in the runtime stage of Dockerfile and gpu.Dockerfile via build args defaulting to unknown, and inject the values from the publish workflows so published images can be mapped back to their source commit.
Add the static title, description, url, documentation, licenses and vendor OCI image labels to the runtime stage of Dockerfile and gpu.Dockerfile.
Add a docker-build pixi task and scripts/docker-build.py that builds the runtime image with the GIT_COMMIT, BUILD_DATE and IMAGE_VERSION build args derived from the working tree, so local images carry the same OCI labels as CI. Point the README build instructions at the task.
Default the docker-build image tag to opendronemap/odm:<branch> when building off master (sanitising the branch name for Docker), keeping :latest and :gpu on master, so local branch builds don't clobber the latest tag.
|
Oh one other thing to note: I think this PR answers the question in #1960 If we use pre-built packages, then we don't run the build through the ODM build wrapper with This would mean we likely drop support for very old CPUs without AVX / AVX2 (as the prebuilt packages will likely include optimised versions for newer architectures). What do you think @MJohnson459? |
|
@spwoodcock It's a good point, there's a lot going on in this PR so thanks for bringing it up. I'll try to explain what I've done and you can check me, I spent a bit of time trying to work this out myself. CondaFor the first piece of the puzzle, Conda compiles any C++ on Linux with these flags (trimmed down for clarity): CXXFLAGS=-std=c++17 -march=nocona -mtune=haswell -O2(I will note the documentation on this is lacking right now) The However, unlike the previous ODM build it specifies the Finally, this applies to the generic SuperbuildFor the second piece, there are still a few libraries we compile because we have forks. Two of these have I've not explicitly set an SummaryThe best-practice for C++/cmake libraries is to omit - Conda sets It also means we are likely missing some optimisations, but its not as clear cut how much the Does that all make (some) sense? |
It certainly does! That's much clearer 😄 Thank you - I should probably close the linked issue then, as I think it's handled well. |
3 participants
This moves the entire ODM buildsystem to pixi. This allows us to use prebuild binaries for our libraries massively cutting down the build time, while also introducing a single pixi.toml file to store dependencies.
With this change, anyone should be able to build and run ODM like this:
This should work on Linux (x86, aarch64), Windows, Mac. GPU support is also available by adding a
-e gpuargument to the commands.Changes
Testing
To ensure this doesn't cause any major regressions, I've been keeping an eye on the following canaries:
Block-Michenzaniandodm_data_aukerman. I have been running these with no additional flags and to my untrained eye the results look pretty consistent.Missing testing
While I have tested the basics, there are some areas I'm aware I haven't tested enough yet:
Future work
pixi.lockfile to lock the exact versions.