Our Build Implementation

Building and Git Submodules

We rely on git submodules in several areas of our agile courseware program. But this dependence can be tricky: while git supports submodules, it often does not seem to "like" doing so! Here we will write up some problems we've hit in that area.

The first thing we are going to document is how to clone from a repository. We need to open our terminal and cd into the file we want to clone our repository in. Once we have done that we run the following command:
git clone https://github.com/gcallah/DevOps.git
(That last parameter is the clone string you get from the GitHub webpage shown below in the second screenshot.)

Screenshot - Cloning a Repository.

Screenshot - Github Webpage.

If you need to add a submodule whether for first time or again, it is actually easier than some of the online document that suggest.For instance, to add TheDevOpsCourse to DevOps, you would just run:

git submodule add https://github.com/gcallah/TheDevOpsCourse.git
(That last parameter is the clone string you get from the GitHub webpage.)

Screenshot - Adding a Submodule

Sometimes a submodule can be corrupted and will have to be removed. This process is difficult !

Screenshot - Deleting a Submodule
Deep Dive Into Our Make File Workflow
Symbolic Link

Symbolic Link or Soft link is nothing but a file that points to another file. In our root directory we have the submakefile and if we look closely at the makefile inside each team directory (build,cloud,monitoring etc) using commands: ls -las

Running ls -las inside build team directory

We see that makefile is referring to submakefile with a pointer (makefile --> ../submakefile).


Following is the command to create a symbolic link:-
ln -s source_file destination_file
In order to create a makefile to perform the build process we will have to do the following in the main directory:-
ln -s submakefile destination_file
NOTE:- The destination_file should not be created in the destination directory prior to running this command. It will be created by itself when we run it.

Submakefile

After taking a look at the Symbolic link we now know that the makefile is referring to the submakefile. The submakefile has the following contents:-

Submakefile code

In the first few lines of the code we have just initialized a few variables to make sure that we don't repeat ourselves. On line 17 we run shell commands to list files (ls) and stream edit them. ls first list all the files present inside the html_src directory which was initialized with PTML_DIR on line 7. Then we use a stream editor on these files to convert from .ptml to .html file and then extracting out the html_src file from it which would lead to placing it in the team directory.The test_html.sh has all the code test required to run on our html files for example html checker, url checker, spell checker e.t.c. The '%' sign acts as a wildcard (*) in a pattern. $< is the first prerequisite, whereas $@ means evaluate all.

Test_html file

Test_html file is basically the file which consists of all the codes. This seperation is done to avoid repeating ourselves since this code may be required by different teams to perform their own testing.

How to implement the build process

Build tools control the building of a final product from some number of pieces that compose that product. The final product could be a web page, a compiled executable, a piece of documentation, or a book. Please Note that the following steps are only for Linux, MacOS. Windows users, please have a look at the bottom of the page.

The key characteristics of build tools are that:

  • They automate the build process.
  • They re-build only what is necessary to re-build.
  • They document the build process.

How to ‘make local’?

If you want to make changes to an html file, do not directly edit it! Each directory(eg. cloud, testing...etc.) has a html_src folder/dir. This html_src folder contains .ptml files for all .html files. So, in order to edit ‘abc.html’, we need to edit ‘abc.ptml’

  • Step 1: navigate to required html_src folder
  • Step 2: edit ‘abc.ptml’ file
  • Step 3:
    git add abc.ptml
    git commit abc.ptml
  • Step 4: cd .. (basically come to outer directory)
  • Step 5: Do you see a makefile? great! :blush:
    make local
  • Step 6: now check your abc.html file. It should have all changes made to abc.ptml
  • Step 7: git add and git commit the abc.html file
  • Step 8:
    git push


Frequently Asked Build Questions
Why is that when I run make local, I get a message saying:
make: Nothing to be done for `local'.

The build tool make takes into consideration the timestamps of ptml (pre-html) and the html. If our ptml is newer than the html files then the command make local will work. If you have been editing the html file and it has a timestamp newer than ptml then you are likely to see this error. One more possibility is when you run make local on a file and without making any changes we run make local again. This means that both the timestamps are the same and our code will assume there is nothing to be done.

How to install Command Line Tool on Mac OS X ?[1]

The steps listed was referenced from OSXDAILY[1] and will guide you through the process of installing command line tool on Mac OS X

  1. Launch terminal which can be found in /Applications/Utilities
  2. Type the following command:

    xcode-select --install

    Terminal will display the following if you haven't downloaded it before:

    x-code select: note: install requested for command line developer tools

    However if you have downloaded it before then you will get the following message:

    xcode-select: error: command line tools are already installed, use "Software Update" to install updates
  3. A simple software update pop up will prompt you to select one of the three options.Options are Get Xcode, Not now and Install. Get Xcode will install both Xcode and the command line developer tools from the App Store. Not now will quit the installation process. Install will lead to installation of the command line tool once you have accepted the "Terms of Service".
  4. The Command Line Tools package may take upto several minutes to download based on your connection speed.
  5. The installer goes ahead smoothly and completes on its own . You can confirm everything is working as expected by trying out the commands like gcc, git, svn, rebase, make, ld, otool, nm.
  6. The command line toolkit package can be found in the following directory:

    /Library/Developer/CommandLineTools/

Sources: OSXDAILY

python 3

Put a symbolic link called ‘python3’ in the directory where you have python: now running ‘python3’ should work. *But* make sure it points to some python 3 version, and not 2.7.

Problems with your git submodules

Make sure you have the all the contents pulled from the submodule if build fails. In order to do this follow the steps shown below once you are in the main DevOps directory.
git sobmodule update
Alternatively we can also do the following: -
cd utils
git pull origin master

Build doesn't work even though you followed all the build steps

You must make sure that your utils folder is upto date. Hence we will have to cd into the utils folder and do a git pull origin master

Cannot build website

You must run that from the "main" DevOps directory, not your team subdirectory. If you only have Python 2.7… time to upgrade!

Sources:

  1. OSXDAILY