Note: This is a republication of an article I wrote a couple of years ago on Medium in 2020.

TL;DR

Limited internet during emigration meant unreliable Docker builds. So I built a local CPAN mirror using minicpan. Powered by 4, no 3, no 2G internet and imagining 56k modem sounds.

You can clone the project at gitlab: gitlab.com/waterkip/docker-p5-minicpan (MIT license).

Stranded on a tiny tropical island

I am in the process of emigration and while I’m doing that I have no access to high-speed internet. I have one 4G connection and a 3G connection that both have a 3Gb limit a day. I get downgraded to 2G afterwards, which is slow. Besides the FUP there are also many times where my connection just drops for no apparent reason. The experience has led me to enter a stage in my life I haven’t seen since puberty, offline working.

Pet project becomes life saver

I write Perl software for a living and needing to build Docker images for my projects is one of the ways I earn that living. Since my internet is flaky and up to a point not trustworthy I needed something that mirrors CPAN locally. A year ago I wanted to create a CPAN-mirror at work to cut back on the downloads from (Meta)CPAN. I never finished that pet project, but I still had some remnants in one of my git repos. The first couple of weeks of living of a 3/4G connection was acceptable, while it has become the standard for over a couple of months, the annoyances of having failed builds because I could not connect to CPAN needed an intervention.

So I did a few things. First thing I did: Gitlab CI for daily building of images that I can use as a base for developing. I’m now building four images for Perl, one with some development tools on it and one without in both a normal and Moose flavor. Thank you Gitlab, U just had to pull an image and we were of to the races.

The second thing I did was a bit more troublesome to get working. A CPAN mirror locally. The CPAN mirror itself was easy to setup, you just need CPAN::Mini and a little configuration file and you are done. Using a volume is easy too. But getting that named volume working is a bit harder.
Long story short, don’t use VOLUME in your Dockerfile and don’t try to become another user other than root. It will make you go mental in a couple of hours. Some people reported that they have made it work. I thought so too. But that joy ended as soon as I removed the complete setup and tried to build it again.

Remember, I was having problems with building Docker images locally, so I used Gitlab CI for building my images and reusing them. Push, wait to build, pull, test out the changes. It wasn’t fun. Don’t use volumes when you want to become a non-root user that uses that volume. This issue on Github proves it

Your own private little Idaho aka DarkPAN

The Dockerfile for your local CPAN is pretty straight forward. Your docker-compose.yml will look a little something like this:

services:
  minicpan:
    build:
      context: .
      dockerfile: Dockerfile
    networks:
      - default
    volumes:
      - minicpan:/opt/cpan
      - ./minicpanrc:/opt/cpan/.minicpanrc
  web:
    image: nginx:latest
    depends_on:
      # It doesn't but firing up, will trigger a execution of minicpan
      # so we are updated as soon as this thing does its job
      - minicpan
    volumes:
      - ./cpan.site:/etc/nginx/conf.d/default.conf
      - minicpan:/opt/cpan
    networks:
      minicpan:
        aliases:
          - cpan.metacpan.org
          - www.cpan.org
    expose:
      - "80"
    ports:
      - "8080:80"

volumes:
  minicpan:
    external: true
networks:
  default:
  minicpan:
    external: true

There are a couple of things that make this usefull when integrating your local mirror with other docker (compose) projects. First of all we have two things that are not maintained by this project, both the volume and network minicpan. These are made by you before you startup your docker-compose instances:

docker network create minicpan
docker volume create minicpan

In the docker-compose.yml file you’ll see that they have external: true, this means that docker-compose will complain and not start if the network and volume cannot be found. We can reuse both the volume and the network in other projects. You just add the same sort of configuration in the other project:

services:
  perlthings:
    build:
      context: .
      dockerfile: Dockerfile
      network: minicpan
    networks:
      - minicpan

volumes:
  minicpan:
    external: true
networks:
  default:
    minicpan:
      external: true

In this example I only use the network, as the volume isn’t needed, but you could reuse the volume for other things. I plan on reusing it for the MetaCPAN docker project (because I want to get that working, so I’ll have a local MetaCPAN as well).

I’ve added some aliases so we know which hosts can be found in a network:

services:
  minicpan:
  networks:
    minicpan:
      aliases:
        - cpan.metacpan.org
        - www.cpan.org

This is needed to allow other docker instances or docker compose projects to reach your machine to use your mirror. This way you don’t need a complicated setup with passing PERL_CPANM_OPTs during the build or during running the container. You just use the standard hosts. Now when another docker container needs to connect to cpan.metacpan.org your mirror is the target.

NginX is simply configured to allow any kind of server name you desire. You can view its contents here.

Now you can fire up the docker instances:

docker compose build minicpan
docker compose up --no-start
docker compose start # or just if you prefer less typing
docker compose up -d

It took me around 36 hours to get all of the modules from CPAN. Compared to the 8 hours with my high-speed connection back in the Netherlands… It was a slow day. You could also opt to not pull in Bio-perl and such, but I felt greedy (see CPAN::Mini's `skip_perl` ).

And it paid off. Now I don’t have build failures anymore it is blazing fast. The updates trickle in when I stop/start the nginx container. Maybe the next step is to start this beauty up at boot. Although as stated earlier, I would like to hook into the MetaCPAN docker project. Because that would be sweet to have. Documentation at your finger tips.

Some small things to consider. After your initial pull of the Perl modules you can even build your own Dockerfile against your own mirror:

services:
  minicpan:
    build:
      network: minicpan
    networks:
      - minicpan

networks:
  minicpan:
    external: true

The mini-meta approach ;)

Some caveats may apply. CPAN::Mini takes the latest and greatest of modules and that is good enough for me. If you want to have older versions of a package you are out of luck. Things on the backpan are not included. When you need to install Debian packages during your build process you are also out of luck, these still require a working internet connection. You could opt to mirror those as well but with projects that have Alpine, Debian and perhaps other base images this might lead to mirror the whole internet to your box and that might be a bit of overkill.

Conclusion

I no longer have build errors because my connection with CPAN times out or other funky things when building my images \o/

And before I forget, just add this to your .zshrc if you want to enjoy it outside of your docker projects:

export PERL_CPANM_OPT=”--mirror-only --mirror http://localhost:8080"

Credits: The NginX configuration is taken from github.com/colinnewell/CPAN-Mirror-Docker