Building the Backend of meldR, an LXCP, with Conda Packages
Data Society developed meldR, an LXCP (learning experience communications platform) for companies to quickly deliver and manage complex data science, AI, and ML upskilling programs.
meldR and its engine consist of multiple services and components linked together through several pipelines. Each pipeline runs within a Docker image, where we spin up an isolated conda environment that includes both standard and custom-built proprietary python packages.
One example is a course_scraper package that takes in a folder with custom R markdown files that include technical content for a course like Introduction to Python. Each template contains the theoretical materials on the slides and the executable code used by the instructor and the students for practice. Our scraper pipeline renders the HTML slides, scrapes executable code into Jupyter notebooks, bundles it all up, and pushes it to a remote bucket from where it will be served to our students via our meldR app and JupyterLab environment.
For all of these things to work smoothly and be reproducible on our local machines and running in our remote engine, we choose conda to handle the heavy dependency lifting and environment spinning.
Custom packages are no different in that sense. When we develop the custom package code and build the package through pip, we push it to our remote artifact registry, where we install the package in the respective conda environment. We manage and track all these dependencies in our conda YAML files, where we note which packages and versions we need for each environment and from which channels and repositories we would like to install them. Most of our packages are standard and come from either default or conda-forge channels. For all custom packages we build with pip, we configure them there and then pass the right package names and versions to our conda YAML files.
The conda environment also allows us to test and install custom packages in the already created environment on the fly. This is a necessary step for us to test any new package code or updates to the existing package locally within the conda environment before it makes it to our artifact registry.
The key to ensuring all pieces fit together is dependency handling and potential dependency conflict resolution, which conda enables us to do smoothly.
Watch video Building the Backend of meldR, an LXCP, with Conda Packages online without registration, duration hours minute second in high quality. This video was added by user Anaconda, Inc. 08 November 2023, don't forget to share it with your friends and acquaintances, it has been viewed on our site 697 once and liked it 1 people.