Category Archives: LearnWith

LearnWith: Concept to Implementation – Docker Development Environment

In this installment of the LearnWith series I create a standalone development environment using Docker and invite you along on the journey of working through the process outlined here. I go from having a vague idea of what I need with very little prior Docker experience, to a working solution. Rather than be a tutorial on Docker, the focus of this article is on the method and reasoning behind getting to a solution. I do present a solution at the end containing all the notes and explanations gained from following the process. So if you’re here for that, this article is not all theory, I’ve got you covered too. Come LearnWith me.

NOTE: If you are here just for the solution, skip to The Implementation section.

Defining the Problem: The Why

I’ve been working on Project ︎◼︎◼︎◼︎◼︎◼︎’◼︎ ◼︎◼︎◼︎◼︎◼︎ for a while now. So far, it has a Front End which is a little baby skeleton of a website and a browser extension; both of which are supported by a data API. The website is a web server running NodeJS, while the API is made up of Elastic Search and another web server running Java.

For the next part of Project ◼︎◼︎◼︎◼︎◼︎’◼︎ ◼︎◼︎◼︎◼︎◼︎, I’m creating an Editing Suite to manage the source data the Front End needs to work. To support the Editing Suite I’ll need a web server, a queue, queue processors and an a second API composed of a web server and a database.

Getting to the problem… Imagine running the full system on your local machine; the Front End and the Editing Suite. A minimum of 8 processes would need to be running on the same machine. That’s a lot of starting and stopping of services and a lot of port management. I currently need half of that and I’ve already started losing track which exact processes are needed for specific parts to work.

This very close to an actual conversation I’ve had when doing an impromptu demo: “… and after the extension is installed you see a list…? Where’s the list?! Ahh, I forgot to start the web server. Refresh and…. hold on, forgot to start elastic search…? Refresh and …Ngggggllff!! I changed the port earlier… Restart and refresh… What’s wrong now…? I’ll show you another time….”.

As any system grows, there comes a point when you need spend time on managing the complexity to continue being efficient.

Fortunately that demo scenario I described was to an understanding fellow dev friend, so not a big deal. So far I’ve not had any significant problems dealing with the complexity. Doubling the number of processes, doubles the complexity of the system, which also doubles the points of failure. As any system grows, there comes a point when you need spend time on managing the complexity to continue being efficient.

Side note: For Devs that don’t have any love for project managers, managing complexity is a part of what they do. They manage the complexity of coordinating different types of people doing different things that are all part of something larger. It’s not easy by any means. Shout out to my PM friends!

To summarise and to state concisely why we’re fixing this problem and what the proposed solution should be working to fix:

  1. Working in the development environment of the app is becoming complex.
  2. Multiple services all use the same port sometimes causing clashes.
  3. Getting a service running is tedious and error prone.
  4. Managing different versions of software for each service is difficult on the local machine.
  5. Versions of software installed on the local machine may clash with software needed for other projects.

Additionally this is going to be used for development, so ideally the solution should not fundamentally change the development process too much.

Defining the Problem: The What

We just described what we don’t want. Let’s describe it in the positive, let’s describe what we do want:

  1. Different parts of the app (the Front End and the Editing Suite) can be run independently.
  2. Simplify the starting and stopping of services.
  3. Every process needed will start up if it’s needed for a service to function correctly.
  4. Each process can have it’s own software version.
  5. Changing a file locally must still change the file represented in the output.

We’ve described what an outline of the solution should look like. This where come up with different technologies and evaluate them based off of these criteria. In an ideal world you’d go with one that has the most merit. In the real world, your situation (money, time, politics, available skills) often guides your choice. I’ve done this before using Vagrant. This time, as the title suggests, I use Docker. For the sake of brevity, and because this isn’t an article on the pros and cons of Docker vs Vagrant, let’s assume an analysis has happened and we’ve verified Docker fills all 5 of the requirements above.

Service Architecture. (If I had a child they’d have drawn it like this. I don’t have a child, so I had to draw it like this myself.)

Referencing the diagram above, we’re getting to the details of the setup we need. A successful implementation would mean we need all the following running in docker containers:

  1. An instance of the Java Application
  2. An instance of MySQL
  3. An instance of RabbitMQ
  4. An instance of the Python Worker Application

Additionally the following constraints would need to be satisfied.

  1. The Java application must able to communicate with RabbitMQ.
  2. The Python application must be able to communicate with RabbitMQ.
  3. The Java Application must be able to communicate with MySQL.
  4. Altering the Java code on the local machine must alter the code running in the container.
  5. Altering the Python code on the local machine must alter the code running in the container.

Note on technologies chosen: The choice of technology to build your project in, is in some cases arbitrary. I chose MySQL because amongst other things, it’s free, open source, stable, and a relational database that matches the requirements of my data. I’ve worked with it before and it’s more than good enough to get the job done. There may be better a database but does it really matter? Everyone has a bias. A few of the biases/reasons I’ve heard are “I know how to do it this way”, “It just feels right”, “Google does it this way”, “We’ve always done it this way”, “I don’t trust other people’s code”, “It’s cheaper and easier to find developers”, “I was here first”. Some of those are valid, others not. The process should be: have some options, build consensus around one or more of the options, choose one, implement it.

I chose Java and Vertx because I wanted to broaden my skillset. I could have used PHP or NodeJS and the result would have be the same. If any juniors are reading this, if someone is telling you your programming language is inferior, they’re likely gate keeping. If you press those people, you’ll generally find in most cases they’ve barely used the technology they’re criticising. Some of the criticisms might be valid, take what they’re saying with a pinch of salt, get a second opinion and take that with a pinch of salt too.

I chose RabbitMQ quite frivolously. I’ve heard it mentioned and it’s the name I’m most familiar with when it comes to queues and queue processing – even though I’ve never used it. That and it’s also open source and free. I might find it can’t do what I need, I doubt it, but if that happens I’ll just swap out the message transfer part for something else.

As an aside: If you find value in using open source software, make an effort to support those communities. Where would the world be without them?

Python arguably has the best support for Natural Language Processing so I chose that for this specific worker. There may be other workers that work better in other languages. Though it might make sense to keep your stack as small as possible.

Ultimately everyone has a bias, I’m showing mine because this is meant to be an honest peak behind the curtains of software development. As I mentioned, your situation can dictate your solution. If I make a mistake on this project, no one will die. With that in mind, my approach is: give something some thought, do that something, maybe make a mistake, learn from it, improve, update the implementation.

Preparing for Implementation

We know what we need to do, but we know nothing about Docker. For us mortals that mostly deal with common problems, we’re usually not the first person to try and solve a problem. And even if we are, chances are someone has done some work on the subject before us and we can build on what they’ve done. Maybe someone has written about it, maybe there are some tutorials, some online courses. All of these things can help us get to where we need to be.

Think of learning the terminology of a technology in the same way you’d ask for directions in a country where you don’t speak the language. You will probably get there, but various misunderstandings might mean there will be a few wrong turns along the way.

The first thing with any new technology is to read it’s introduction. This will give us a brief understanding of what we’re dealing with and the terminology we need to speak about our solution. The terminology is important because it gives us a way of speaking to the wider community allowing them to be able to understand and help us. Think of learning the terminology of a technology in the same way you’d ask for directions in a country where you don’t speak the language. You will probably get there, but various misunderstandings might mean there will be a few wrong turns along the way.

Docker’s overview page seems the perfect place to get this good initial understanding. Get started with Docker looks like a good next step. And a quick browse of the menu shows a Develop With Docker section. This looks promising! The official website looks like it has all the information we need to get to a solution. I can’t stress how happy this makes me: All or most of the knowledge will come straight from the source; The methods used will be officially supported known ways of doing things; And people already versed in the technology will be more likely to follow the implementation. All good things!

Now to read through the links and run the tutorials posted above. Depending on the time I have available, it’s likely a few weeks will have passed before I get to the next section.

Completed the Readings

I’ve been trying to read during my lunch break and after work. 2 weeks have passed. It would have taken me longer but I’d already gone through 60% of the readings before starting this write up. Most of it was a refresh.

I’ve got lucky with some very good documentation. I’ve read the overview and completed the getting started tutorials. I have enough information to start! I’ve not read the develop with docker section. I’m probably missing a key piece of knowledge to solve the problem completely. So… enough information to start but not enough to finish. I prefer starting as soon as I can. Others may prefer to do more reading. In this case, I don’t to have worry about the piece of string issue because I have already defined the problem. As long as I have that goal in mind, the work I do will generally get me closer to a solution. I also know from the readings, that anything I’ll need to change won’t need significant effort or have significant repercussions.

As I mentioned previously, this isn’t meant to replace the well written tutorials on the web, rather an exploration of the method of reaching the solution. With that in mind, onto the implementation.

Onto the implementation!

The Implementation

Note: If you want to see a step by step evolution of how I got to the solution, take a look a the source code on github. Look through branches ‘Part 1’ through to ‘Part 7’ to see the incremental steps needed to arrive at this solution.

You’ll need to add two files to your project to get it to. A Dockerfile file to tell docker how to package up the code and how to prepare the environment the code needs to run in. And a docker-compose.yml file to tell docker what other services are needed for the app to run and how to run them.

The Dockerfile:

FROM openjdk:8u212-jdk
COPY . /app
RUN ["./gradlew", "clean"]
CMD ["./gradlew", "run"]

Lets look at this line by line:

FROM openjdk:8u212-jdk
This tells Docker to base my image off of the official openjdk image. This means I will have a specific version of the Java OpenJDK installed before I even begin. Docker will try to find this image on the local machine. If it can’t it will try download it from the docker hub. There are loads of official images on the docker hub you can use in your app.

This tell Docker that the commands following this one will be run in the container from the directory /app.

COPY . /app
This tell Docker to copy everything in the current directory (denoted by the dot) to the folder /app on the container.

RUN [“./gradlew”, “clean”]
This will tell Docker to run the java specific command ‘./gradlew clean‘ (from the previously set WORKDIR) – I run this to make sure the gradle wrapper is available on the container before packaging it.

The Java application runs on port 8888 and this tells Docker to allow access to the application from via this port.

CMD [“./gradlew”, “run”]
CMD is the command Docker uses to start up your application. In this case to start my Java application Docker will run “./gradlew run” from the WORKDIR

Docker will now know how to build your code into an image and have it be able to run on it’s own.

Next docker-compose.yml:

version: "3"
    build: .
      - './:/app'
      - "4000:8888"
      - mysql
      - java_to_db
      - java_to_queue
    image: mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
      MYSQL_ROOT_PASSWORD: example
      - java_to_db

If you’ve not seen yaml before, it’s a text file format where the number of space before the text has meaning. If something has more spaces than the line before it, that means it is relating to the line before it. So in the example java_app and mysql are sections that belong to services. Let’s continue line by line:

version: “3”
Tells docker what version of the docker file structure we are using. Different versions have different sections and capabilities.

This section defines the list of networks and lets Docker know how to set up these networks. A network is used to allow communication between services and the outside world. The lines below network ‘java_to_db:‘ and ‘java_to_queue:‘ tell Docker to set up two networks with the respective names.

Under this section we put all the process we need to run. MySQL, python, Java, RabbitMQ would all be defined under this section.

This is a name for the service. The name itself can be anything and has no specific meaning. However you may need to refer to it in your application so make it something meaningful.

build: .
This is tabbed into the java_app section meaning it’s a command for the java_app. This tells Docker to build run all the instruction in the Dockerfile in the current directory. (the one we defined above).

We’ll do volumes and – ‘./:/app’ together. Volumes is part of java_app and volume usually means refers to data storage such as files. – ‘./:/app’ falls under volumes. So, with these two lines we’re defining the file storage for the java app. You might remember ./ and /app from the COPY command in the Dockerfile. The copy command copied the all the code at the time the command was run. If any changes have been made since the code is now out of date. – ‘./:/app’ is called a bind-mount and tells Docker that we want to have the current directory in place of the /app directory on the container. This means changes made in the local directory linked in the Docker container too.

Also part of java_app. This and the line below, – “4000:8888”, map port 4000 to port 8888. What this means is that by visiting port 4000 on the (virtual) hardware it runs on, Docker will make sure all data that transfers through 4000 to port 8888 on the container.

This section is used to determine the order in which to start the services. The line below depends_on tells us that the java_app should wait for the mysql service to be ready before it starts.

Sitting under ‘java_app‘ this tells Docker java_app needs to talk to everything in the java_to_db and everything in the java_to_queue networks. ‘java_app‘ won’t be able to communicate with anything anything else.

This is the start of a new MySQL service. The service has the name mysql but could have been anything. I mentioned earlier has is important. In this case when trying to connect to MySQL from the Java App, we need use the service name, ‘mysql’, as the hostname.

image: mysql
This is tells docker to create a service using the official MySQL image.

Other than ‘networks‘, the rest of the commands in the mysql section setup mysql to run in a specific way, you can find more details on the official MySQL Docker image page.

And that’s it. We’re finished! That’s everything needed to a development environment running in Docker.

Reflecting on the Goal

Are we finished? Let’s look back at what we set out to do. Let’s look at what we have:

  • An instance of the Java Application
  • An instance of MySQL
  • The Java Application must be able to communicate with MySQL.
  • Altering the Java code on the local machine must alter the code running in the container.

And what we we still need:

  • An instance of RabbitMQ
  • An instance of the Python Worker Application
  • The Java application must able to communicate with RabbitMQ.
  • The Python application must be able to communicate with RabbitMQ.
  • Altering the Python code on the local machine must alter the code running in the container.

That looks like more than half of the original plan is still missing. How in any way is this finished?

The Business Answer: Priorities change constantly when working on a project. Things outside of your control often affect what you’re doing on the project. Perhaps something your boss asked for, perhaps something a competitor has done now means something very important yesterday, has had to be moved out of the way to make place for something else today. I purposefully put in an extra bit to show how a modular iterative approach to solving a problem can mean that even though we’ve only done a piece of the work, the solution in it’s current state still helps us move forward. If we come back to the solution later, we can pick up where we left off.

The Truth: I’m currently the only developer building the proof of concept for Project ◼︎◼︎◼︎◼︎◼︎’◼︎ ◼︎◼︎◼︎◼︎◼︎. It’s taken a long time for me to write this article. I’m doing all this while still having a full time job, so it’s taken maybe a month or more. On top of that, I’ve not allowing myself to continue with the code until I finish this documenting the solution. Having the solution means I am unblocked and I’m REALLY excited to get back to writing code again! Having said that, I feel like I have solved the whole problem conceptually. All the core concepts needed to add Python and RabbitMQ are already included in the solution. Adding them in will a copy and paste job with a little bit of reading to get the specifics of RabbitMQ. If there was another developer waiting on the full solution, I’d definitely have made sure it was complete.


Don’t be discouraged if you get stuck. Take a break and come back to it later. While writing I had two moments where I got stuck and despondent. My internal thoughts were: “The solution isn’t doing what I want! I should quit!”, “Is Project ◼︎◼︎◼︎◼︎◼︎’◼︎ ◼︎◼︎◼︎◼︎◼︎ even worth doing?”, “If I can’t even do this small, how dare I write an article and to give others advice?”. Most people I’ve worked with think I’m a good software engineer. To be honest, I am happy with my skills and I do have my moments but I’ve never ever felt like “one of the best developers I’ve worked with“. If that person is right and I am a good developer, I say this to let you know if you’re feeling discouraged, just remember, we all feel that way sometimes, just keep at it.

That’s it! Thank you for reading. I hope you learned something with me and had fun doing it.

LearnWith: Practical Problem Solving

Practical problem solving (Illustration by myself)

In this first LearnWith in the series, I outline my general approach to practical problem solving. In it I detail how I go from knowing nothing to knowing what needs to be done. You might notice I didn’t say “go from knowing nothing to knowing everything”. Knowing everything, although rewarding, almost always isn’t the goal. The goal is to solve the problem.

If you’re trying to know everything about a vast knowledge base before making an attempt at solving a problem, you may be falling into what is called analysis paralysis. We’re getting ahead of ourselves though. Analysis paralysis is usually an issue for someone who has somewhat of an idea of what they’re dealing with.

Define the Problem

If you find yourself worrying and doing nothing or not knowing where to start, you probably haven’t defined the problem or defined the problem well enough. Defining the problem is the first step to solving it. It’s important to define what success looks like right at the beginning. It’s how we know we have finished and we can celebrate a job well done.

The BBC Horizon documentary “How Long is a Piece of String?” can be used as an example why we need to define success. For a seemingly simple task such as measuring a piece of string, we might end up doing measurements in imperial when it might have been needed in metric. Alternatively the measurement might not have been accurate enough, it was measured to the nearest centimetre instead of millimetre. Perhaps nanometres or smaller is the level of accuracy needed. We then need to keep in mind that at a higher levels of accuracy, the room temperature has a big effect on the result of the measurement. What temperature should it be measured at? 0 degrees? Celsius, Fahrenheit or Kelvin?

As you can see, there are MANY possible results to be gotten from measuring a string. When finding a solution, it’s important to be practical. Measuring the length of the string to the nearest cm is much less effort than measuring to the nearest nanometre. The first needs a tape measure and the other needs a laboratory with expensive machinery. Should we do both? If you need the more accurate measurement, the first example is wrong; if the coarse measurement is good enough, the second example would have wasted a lot of money. Both ways measure the length of the string, both provide a correct answer, but is either answer useful?

To define The What of success in a practical way, you need to understand The Why. Understanding why you’re trying to solve a problem gives you the context needed to understand which solutions bring you closer to the result you want. In the string example, let’s say we’ve found out that we’re trying to provide a friend of a friend with a new shoelace. This means we won’t need the expensive laboratory equipment; the tape measure method should suffice. Understanding how the string’s length was part of a bigger goal, we got enough context to rule out the expensive method.

So far so good. However, the problems we deal with are never as one dimensional as measuring a piece of string. What about more complex problems?

Breaking it down

Achieving a complex goal can mean the implementation will need a non-trivial solution, one that requires multiple complex moving parts working together in a harmonious way. If you’re struggling to understand the problem or keeping the whole problem in your head all at once, it’s a often a sign you need to break the problem into smaller discrete problems. Even if the problem you’re working on is not that complex it’s usually a good idea to break your problem into smaller pieces. Breaking your problem up allows you to focus implementation on one aspect at a time with the knowledge that all other aspects of the problem will be done later or, if you’re working in a team, by a different team member.

Once you’ve broken each problem into smaller chunks, you can start solving each smaller problem, keeping in mind the context of the larger problem. This should see you ultimately solve the bigger problem altogether.


Things don’t always go as planned. You often won’t be able to do things the way you’d intended. This will force you into finding a creative solution that might modify how that solution fits into the larger context. If and when you get new information, revise how it affects the overall goal.

Let’s assume a goal of baking a lemon cake. The ingredients are 1 cup flour, 1tsp baking powder, 1 egg, 10ml lemon juice, … etc. You’ve started putting the ingredients together but you realise you’ve bought baking soda instead of baking powder. The shops are closed so you can no longer buy the baking powder, how will you get the cake to rise correctly?! Fortunately, you’ve remembered something from baking school, you remember baking soda and lemon juice can be substituted for baking powder. Crisis averted, but will extra lemon juice make the cake too sour?

In the example, we came up with a creative solution that solved one problem by being able to get the cake to rise but potentially causes another by changing the flavour. You now need to decide if this affect the overall outcome and what further steps need to be taken if any. Maybe add less lemon juice so the flavour stays the same or maybe you prefer a slightly tangier cake.

Try and solve the ‘difficult’ parts first. If your solution hinges on a something that you aren’t sure is possible, get an answer to that first. Do this by breaking down that part into the smallest piece possible, make a prototype or a proof of concept, and get the answer you need. This way you’ll know early on if what you’re planning is viable. Take your learnings and build from there.

Although it’s good practice, you don’t need to understand the full solution upfront. If you’ve broken the problem down in a discrete and self-contained way, you shouldn’t need to understand the solution to all parts of the problem before starting. Often times working on different areas can inform how to approach related but separate parts of the problem. Other times taking a step back by doing something else can give you insight you didn’t have before.

Next in the LearnWith series, I will take specific problems I am dealing with and apply what I have outlined here. By the end of that of that hopefully we will have learned something together and you will have seen how I got to that point.


Twitter is a weird place. I see so many people sharing things and appearing to be amazing. By the way, I choose the word “appearing” not to detract from their achievements but because appearances are not always what they seem. I have dealt with that topic and my feelings around them before.

My feelings now, are around wanting to share and wanting to help but of not being sure if what I have is share worthy. I’m probably selling myself short but I feel like I don’t have much specialised knowledge to share. After 13 years in the industry doing everything I don’t see how this can be case but it feels like whatever I can say has been said before and probably much more succinctly.

Having said that, I believe you should focus on your strengths when trying to do the best you can. The one thing I have been told and has been implied at me on many occasion, is that I excel at is picking things up quickly.

This is, I think, is what I can share. I can share with you the journey of how I learn these things and the findings and conclusions I come to along the way. I can share with you my my process and my reasoning while trying to solve a problem. Apparently one learns better when one is trying to explain something to others, so this process will help me too. Over the next few years while I bring my project to life, I will be tackling many things I have no idea about and I will be writing about them here. We’ll be learning together. I invite you to come Learn With me.