Developer tool pipeline

To develop IT systems efficiently you need a bunch of tools.

Generally speaking the tool set of the developers either is a corporate wide decision or a group decision among a group of developers that work together.

This is an attempt to walkthrough some of the developer toolset.

To make things worse a lot of the components in this walkthrough also come together in bundles with other names, like GitLab, BitBucket and so forth.
I still think it's worthwhile to read up on this so when you google a specific name you'll be able to understand what parts of this it maps to.


Keywords

Keyword Description

The IDE - the coding program

Developers use IDEs (Integrated Development Environment) to edit code. IDEs can have several files open at the same time - often all the files in a coding project.
It also keep track of dependencies, has auto-completion and other code tools as well as profilers and debuggers.

IDE Screenshot
Sample IDE screenshot (Visual Studio)

Examples of features of an IDE:

Most IDEs also integrate seamlessly into version control systems for ease of branching, merging and checkins, debugging tools, and CI feedback.

Many developers spend a lot of time in the IDE since it is their main tool to work in, and hence they tend to tweak it to their will with heavy customization in terms of plugins, coloring, menu choices and so forth.

Common IDEs include:


Version control systems (VCS)

Code consists mainly of text files. It's relatively easy to find differences between two text files and to keep track of what files are changed and not.

Version control systems makes sure each file is managed in its latest version, and that you may change back to a previous version of a file, or a set of files, if you want to.

Since version control systems work on the text file level of the project they typically don't care about what programming language you use. Most VCSes can be used with any programming language and any preference is rather derived from out-of-the-box integrations in language specific IDE:s

Version control systems are more than a backup feature of code. It's also the tool that help with merging upon code conflicts, branching for ease of development, communication of changes (through commit comments). It's no wonder many developers are quite vocal about their preference of version control tool.

Typical version control systems include:

Merging

If two persons are both downloading version 1 of a code base to their computer and keep working each on their feature there is a risk that when they check their code in to the main branch both have made changes to the same files that end up in conflict. This will trigger a merge request.
Many modern version control systems resolves many merge conflicts automatically, but whenever it is unsure of how to solve a conflict manual merging is required.

Branching

Version control systems also allow for branching. Say you have a stable version of the code base but you want to start implementing a heavy feature that will take some time to implement and will include changes to a lot of files in the code base. If you start doing this change in the main branch you will have trouble making smaller changes to the main branch and instead you opt to branch out a separate branch of development.
This way you can still make swift minor changes to the main trunk and concurrently keep working on the separate branch.

Merge of feature
Picture from https://oer.gitlab.io/oer-courses/cacs/Git-Introduction.html#/slide-git-merge

When it's time to introduce back the changes from the feature branch to the main branch you often end up with a merge.

Graphical version control tools

Some people prefer to be able to look at version control history as a graph of commits and branches. A lot of tools exist to help with this. Each version control system has a range of tools for that exact technology.


Code dependency management

For efficiency when writing code you often make use of code others have written. It's often widely used public libraries available.
Dependency management resolves conflicts when different versions of the same dependency are called for.

For example, say you are writing a service that expect data posted to it in JSON format. It would be tedious to write your own JSON interpretation libraries, so you probably end up using for example Newtonsoft JSON libraries. It makes sense. They are proven to use by millions of usages, it's easy to find usage examples and support on the internet and you will not have to go through the long trial and error period of encountering every JSON obstacle there is.

The dependency management helps you search for relevant libraries, helps your code project keep track of references to these dependencies, downloads them at runtime, enables the content of these libraries to be used in your IDE, caches the libraries and so forth.

Now say you also use another library for logging purposes, and this logging framework has a feature of exporting errors as JSON and that this library is using the same library as you do - but in an older version.
The dependencies management now makes you aware that you have conflicting dependencies and tries to resolve them.

Transitive dependencies

Transitive dependencies are the dependencies of the dependencies you are referring to in your code. These are potentially problematic since an upgrade of a package your code depend on might include upgrades of its dependencies, and those might be incompatible with other dependencies you use.

There are a number of ways to manage transitive dependencies, but they tend to come with the cost of bloated libraries with a lot of security attack surface and longer time to test, deploy and performance degrading the system.


Binaries repositories

A code project consists mainly of text files. It is first when the code is built that it become binaries (some programming languages, like javacript, bash or Python, are interpreted at runtime and never become binaries).

It's sometimes more efficient to deploy the binaries rather than building the code towards different environments. Many industries also has compliance requirements that the binaries used in production should be digitally signed. Since the binary files are the significant ones it makes sense to have them in a separate repository.

Some organizations find it relevant that the development is only performed against tested and proven versions of dependencies. They make it hard to retrieve libraries from the Internet and set up an internal proxy where dependencies are retrieved from rather than allowing full access to internet sources.
To do this they setup internal nuget servers, nexus servers or Artifactory services.


CI/CD pipeline

The process of creating, testing, and deploying a software system from code include a lot of different steps that often depend on each others outcome.

When automating this process some kind of step engine is needed. That is the CI/CD server.

The CI/CD server notices when someone checks in changed code, and then starts to execute the sequence it has been setup to run. If any step fails the rest of the run could be aborted.

A sample CI step flow could look like:

  1. Identify that a code change has been checked in
  2. Clean local build folder
  3. Fetch the new code base
  4. Retrieve the dependencies given by the code project
  5. Build the code
  6. Produce documentation
  7. Append license text to binaries
  8. Compress binaries
  9. Initiate monitoring during testing
  10. Run unit tests
  11. Propagate build status to information radiators
  12. Deploy binaries to integration test environment
  13. Spin up the system on the integration test environment
  14. Setup test data for integration tests
  15. Execute integration tests towards the integration test environment instance
  16. Sign binaries and deploy them to artifact storage
  17. Deploy binaries to system test environment
  18. Spin up the system in system test environment
  19. Prepare the system test for test
  20. Execute system tests
  21. Report results

Some of the steps in the flow above would certainly depend on the outcome of the previous one. For example it's no use to continue if the unit test fails. Then the results should be reported back immediately.

Common CI tools include:


Appendix: Table of common development tools

Code editors (IDE)Developers use IDEs (Integrated Development Environment) to edit code. IDEs can have several files open at the same time, keep track of dependencies, has autocompletion and other code tools as well as profilers and debuggers. Version control (code and file resources)Version control systems keep track of changes to files. They use different merging mechanisms to manage that several people may have changed the same file concurrently. Files are generally managed as a group of file with the same version. Code dependency managementFor efficiency when writing code you often make use of code others have written. It's often widely used public libraries available. Dependency management resolves conflicts when different versions of the same dependency are called for. External dependencies proxies/cachesFor ease of management (security, versions) many organizations use a proxy for binaries. Artifact managementFor ease of management (security, versions) many organizations use a proxy for binaries. CI tools (pipeline engines)Tools to drive the CI workflow and dashboards for results.
C#C# is a Microsoft programming language that is compiled (with minor variations) for .NET Framework or Net Core Framework. The common denominator is compiling for NetStandard wich will make it work for both.
Microsoft Visual StudioHeavy but versatile and competent IDE TFS VCThe build in version control system from Microsoft. Nowadays often substituted for git. Nuget ManagerNuget packages (pronounced as "nu-gets") are zip files of binaries. They can be multi-target and consist of several versions for different frameworks and framework versions to support transitive dependencies for different framework versions. Nuget Server ShareIn its basic form a nuget server is just a file share. TFSTFS is a multi-purpose group software for development. It consists of code repositories, CI/CD engine, task tracker and information management tool.
JetBrains RiderJetBrains provides developer tools. The C# version of their IDE is called Rider. SourceSafe Azure DevOps Resource managementAzure is the online version of TFS and it is a multi-purpose group software for development. It consists of code repositories, CI/CD engine, task tracker and information management tool. Azure DevopsAzure is the online version of TFS and it is a multi-purpose group software for development. It consists of code repositories, CI/CD engine, task tracker and information management tool.
JavaObject oriented, Oracle owned. Open Java project carries an open source version. Object oriented, strict in an academic sense, and hard typed.
EclipseVery extensive IDE with thousands of plugins. Boring look, and intimidating in features. mavenmaven is a life cycle project tool for Java. It manages dependencies in a hierarchy of project modules, it states the build order and it executes tests among other things. Built around a plugin structure with loads of 3rd party plugins. nexusNexus is a local cache of maven packages that would othervise be fetched from the internet. Used for performance and security reasons as well as for internally developed packages. mavenmaven is a life cycle project tool for Java. It manages dependencies in a hierarchy of project modules, it states the build order and it executes tests among other things. Built around a plugin structure with loads of 3rd party plugins. mavenmaven is a life cycle project tool for Java. It manages dependencies in a hierarchy of project modules, it states the build order and it executes tests among other things. Built around a plugin structure with loads of 3rd party plugins.
JetBrains IntelliJCompetent IDE. A little more easily used than Eclipse, but not as feature-less as Notepad or NetBeans. gradleMore modern YAML based substitute for maven (maven configuration is XML based) that also provides programmable parts.
Javascript, Python, and others (agnostic)A lot of different programming languages uses the same type of tools.
Visual Studio CodeMore light-weight than the full Visual Studio but with a myriad of plugins. gitA distributed version control system. Every person has a full setup of the full history of all files in the repository on their machine. Very popular. Very robust. ArtifactoryA repository manager for any type of file but mainly used for binary files. Good meta-data-structure for each file as well as dependencies resolution. File sharesNetwork shares (Windows Shared Folders, CIFS, Samba, DFS, and so forth) Team CityEnterprise level CI/CD engine
Notepad++Only a limited range of full blown IDEs ClearCaseRarely seen nowadays File shares Artifactory JenkinsOpen source CI/CD engine