Introduction
Over the last year or so I have taken a sabbatical from working on other people’s projects to work on my own, and while I have discussed them in other fora, it seems sensible for me to put them down here. As it happens, I have various people interested in my recent work anyway, so it’s about time I did some proper scribbling.
My most recent project, which consumed around six months of full-time work, is a system that I’ve code-named Missive. Missive came out of the observation that beginner programmers (usually using PHP) often report various woes with sending email from a LAMP server, usually for website contact forms. Unfortunately, when I see their code, it is usually susceptible to all manner of security issues, and is best not put on the internet at all.
My proposed solution is a form designer, presented as a wizard-style UI in a web application, which can deploy a secure and extensible form-based PHP application to a LAMP server. For such an apparently trivial problem, the amount of configuration and remote system detection required is remarkable, which might explain why this problem is so daunting for non-experts. I plan to offer this on a free-tier basis to start with, and then add freemium features at a later stage if there is demand.
I have gotten this system to a prototype stage. What follows is a mainly technical walk-through of the system; I hope to do a treatment of the business/entrepreneurial aspects at some other time.
The prototype is on the web, and I’ve also made a start on gathering user feedback. Some documentation is available. I’ve also received some excellent (offline) notes from a UX expert to help me de-tangle my (currently confusing) UI.
User interface
A good start to introducing the project is to examine the visual frontend. As it happens, I have a set of browser tests that do all sorts of end-to-end checks, and so it was trivial to extend that to render each screen and stitch them together. The rendering isn’t great, since it’s PhantomJS rather than Chrome/FF, but it suffices admirably (it’s a very wide image, so readers will probably need to click to zoom):
Screenshots of the wizard stages
The idea is that users will click on the tabs at the top, in turn, and complete each section until the tab turns green. In the right-most tab, their email/form application can be deployed at the click of a button. I suspect some of the UI presentation (and security mechanics) will change a fair bit, given the feedback I have already received, but nevertheless, this is not a bad start.
The frontend uses jQuery with Bootstrap, but perhaps at a later stage, I might look at React or a similar component JS framework.
Micro-services
One of the things I’ve gotten into recently, and which I am quite enthusiastic about, is splitting up an application into smaller pieces (“micro-services”) that are easier to test, replace and repurpose. I’d written a micro-service for a fintech company I was working for (in Node, no less), and I was curious to see how I might build a whole micro-service architecture myself (in PHP, which I am happy and comfortable with).
The following diagram gives a flavour of the system components. It’s a bit out of date, as some of the later services were added when the design crystallised in my mind; nevertheless, it’s still a good structural representation.
Microservices architecture
The system is based on Docker, featuring 12 containers at runtime, 15 at test. It is run atop Docker Compose, which does custom networking between containers, based on what services need to see each other. I use AWS for compute, CircleCI for continuous integration, BitBucket for Git hosting, GitLab for container image hosting, and PaperTrail for log aggregation. I might move to a hosted Mongo solution, but presently self-hosting seems OK. Deployment is a simple manual process at present, but I’d like to move it to Swarm or Kube if possible.
To expand on this a bit, it might be interesting for readers to understand what the pieces do. Here is a précis, indicating what purpose each component has, and how it fits in with my continuous integration and test strategy:
| Name | Runtime service? | Test service? | Library | CI? | Description |
| App | ✗ | ✗ | Core Docker Compose files, an idle shell for hot-swapping services using Traefik, some Traefik configuration stuff | ||
| BoxOfTricks | ✗ | An HTTP API to spin up various servers for testing, currently in development | |||
| Captcha | ✗ | A library to render and verify a CAPTCHA image | |||
| Controller | ✗ | ✗ | A backend server containing HTTP and Web Socket listeners | ||
| Dashboard | ✗ | Presently just a system showing CI build health badges, but will be expanded to show interesting meta-data from around the system | |||
| Integration | ✗ | ✗ | Browser and e2e tests for the whole enterprise | ||
| Interface | ✗ | ✗ | The frontend web app and a web server | ||
| Investigator | ✗ | ✗ | Stores the deployable apps, builds them as zipfiles, and makes them available as a Docker volume for services that need them | ||
| Logger | ✗ | ✗ | Collates log data and pushes it to a remote logging service (e.g. PaperTrail, Loggly, etc) | ||
| MailHog | ✗ | The email testing and API system MailHog | |||
| Mongo | ✗ | ✗ | A Mongo server for data persistence | ||
| MonitorProxy | ✗ | ✗ | A password-protected HTTP proxy to connect to a Netdata instance on the Docker host (this can actually be merged into the Traefik configuration) | ||
| OutTray | ✗ | ✗ | An SMTP email exit point | ||
| ReactServer | ✗ | Some web socket server utility classes based on ReactPHP | |||
| Speaker | ✗ | A simple HTTP library for PHP with a fluent interface | |||
| Storage | ✗ | ✗ | An HTTP API that is required for all services that want to write system state (this is the only service that is allowed to connect to Mongo directly) | ||
| TestBox | ✗ | ✗ | A simple LAMP box used in integration testing (hopefully to be replaced by BoxOfTricks) | ||
| Traffic | ✗ | ✗ | The frontend proxy Traefik | ||
| Transmitter | ✗ | ✗ | A queue-based system to carry out remote investigation operations e.g. checking FTP/SSH/SMTP connectivity, deploying packages | ||
| Utils | ✗ | ✗ | Various low-level PHP libraries used by more than one repo |
Components marked as “CI” may run unit/functional tests, build a Docker image, or a combination of both.
My dashboard is a bit bare at the moment, but still useful – it provides an at-a-glance health check of the CircleCI builds:
Build badges
Testing
The testing of this project is something I’m rather happy about, and I am hoping it will help with some future architectural changes that would otherwise be very complicated to manage. I’ve added tests in a pragmatic fashion where I think they were most needed, in particular to the Storage, Investigator and Transmitter projects.
The Integration container is the last stage in testing, and is scheduled for two daily runs. This pulls the latest images from GitLab and checks pretty much everything from a browser perspective, using PhantomJS with a PHP driver to connect it with PHPUnit. It even checks emails, FTP connections, SSH connections, etc – Docker Compose makes it very easy to bring in extra containers for a complete e2e test. It only takes around 10 minutes to run, so the improve/test loop is kept pretty tight.
Future development
The development effort for this project is managed from a Trello board, and to say the board is full is an understatement. I practised an Agile-lite approach with this project, when it was my priority; I worked out what could be achieved in a week, and then tried to stick to the plan. However, I am now seeking full-time work, so it is rather on pause. I hope to pick this up again soon.
Here are some of the things I’d like to do with this project:
- Upgrade the e2e tests in Transmitter to use external SSH/FTP/web servers in the new BoxOfTricks project
- Implement a new deployment mechanism that does not need web server passwords (to tackle a substantial perception/trust issue)
- Redesign the UI after some excellent feedback
- Swap jQuery for something a bit nicer, maybe React/Redux
- Add some new features to the deployable app, such as a Captcha device