In Part 1of this series I talked about our requirements for the Maven process and the issues we needed to resolve.
In this part I will focus on the infrastructure that Maven needs to have in place and how we set it up to suite our needs..
Repositories
With Maven you use repositories to store your libraries. In fact you store artifacts in the repositories. We have a few repositories for different types of artifacts:
- Public libraries are kept in our public repository.
- Contributed libraries are in the contrib repository.
- Binaries and javadoc of closed-source libraries are released to the public repository and sources to private repository.
- There is a special repository for the third-party libraries. Mainly this serves as a convenience. We do not deploy all the third-party libraries we depend on here but only things we can not find in other public Maven repositories. We also deploy the source code and the javadoc of some libraries so it can be easily attached in IDEs.
- Everyone who uses Maven seriously, is familiar with the pain of coping with SUN libraries. To make this easier on our own developers we have a restricted repository devoted to these libraries. Giving our DevNet teams and customers access to this repository would make their lives easier but would be against the license terms so we can not do that. Nevertheless this is yet another type of library we have a Maven repository for. Note: Java.net provide a maven2 repository but it does not have all of the needed Sun libraries (e.g. activation, mail etc.)
Latest update:- connected archiva to the Java.net maven1 and Java.net maven2 repositories
- configured a nice feature in Archiva that translates maven1 artifacts to maven2 artifacts transparently
and our customers and DevNet guys will be able to avoid problems with Sun libraries now.
- There also are public-snapshot, contrib-snapshot and private-snapshot repositories that host ‘snapshots’ for the corresponding types of libraries. (Snapshot is a special version that indicates a current development copy. Unlike regular versions, Maven will check for a new SNAPSHOT version in a remote repository regularly. Snapshots are also handled entirely by maven thus reducing the management overhead of maintaining and publishing development versions.)
All those repositories are accessible via WebDAV. Here is how access restrictions to those repositories are configured (the arrow pointing back at the repository means write access):
Proxies
We do not use our repositories directly though. Instead – we have two Archiva proxies.
External proxy is hosted outside the office. The internal proxy is in Sydney office. When connecting to m2proxy.atlassian.com internally, internal DNS server will resolve it to the local proxy. Outside the office, it will resolve to the external proxy. This is quite convenient since it allows people to seamlessly roam between offices or work from home without affecting their Maven settings.
Since we wanted only to maintain one proxy for developers, customers and contributors, we have partitioned the proxy into two parts.
- The Public section of the proxy is accessible to everyone and it lets you lookup Atlassian public and contrib libraries, their snapshots and whatever third-party libraries there might be on it. It is also connected to the ibiblio server in case it needs to resolve a new third-party artifact.
- The Private section of the proxy requires authentication (which Maven will handle) and resolves against the above the public one and also against the private and the restricted repositories I described earlier.
The internal proxy, in turn, will resolve against the private section of the external proxy.
The only time when repositories are accessed directly is when writing to them. Write requests are authenticated against our Crowd instance and a distinction is made between atlassian-staff and atlassian-devnet roles for authorization.
Maven proxies also cache things that they download. This means if a version of a product was built once against the proxy, it will always be possible to build that version again even if any one of the repositories is down. This is especially important when it comes to the external repositories. Our proxies are also connected to a few different external repositories in case there is an outage on some of them.
Setting up your local Maven
A Maven environment is configured via its settings.xml file. For our Maven users we have three types of settings.xml.
- Developer settings.xml is configured to use the internal part of the proxy as its central repository.
- DevNet settings.xml is configured for the public part.
- We also ship the DevNet settings above to our customers.
The first two files have to be configured with your username and password if you are going to deploy any artifacts to our repositories. Customers use the repositories for building our products only and normally do not have usernames in our Crowd database.
Summary
The infrastructure I describe above achieves the following:
- Our builds will always be able to find all the dependencies they ever need because our proxies will keep the artifacts forever.
- We do not directly depend on external repositories which makes our builds a lot more robust.
- For the Sydney team (where the majority of our developers are), access to a repository is significantly faster since we have a local proxy.
- There is one set of settings our developers have to maintain in their local environment regardless of whether they are in the office or outside.
- We are less affected by the quirks of the external libraries (legal issues, missing artifacts, etc.)
If your environment is small – you might not need such a complex repository set up. Stay tuned though as the next part is where the actual Maven setup is covered.
.. to be continued
Part 3 will describe you how our projects (all of those libraries I talked about) are configured to make their builds deterministic and repeatable.
Unfortunately, even after several years, our Maven infrastructure is not perfect. Part 4 will cover issues we hit every now and again and explain how we deal with them.