August 17, 2017

Experiments with Packer and Vagrant

  —A remedy for SCP errors with Vagrant and Packer.

To remedy
Build 'virtualbox-iso' errored: Error uploading VirtualBox version: SCP failed to start. This usually means that SCP is not
    properly installed on the remote system.
Install openssl-clients for Linux guests. Doh.

August 11, 2017

Blog Entry Syntax Highlighting

  —A method for syntax highlighting using the Pygments Python module.

I've had good success using Pygments to highlight code on this blog. It provides the best of both worlds: a large selection of language support and it generates standalone HTML.

Standalone HTML was important because many of the other solutions relied upon java script and some of these scripts are hosted on external servers which increased the possibility that the highlighting would disappear whenever the server was down.

I use:
pygmentize -f html -O style=emacs,linenos=1 -o test.html cmd.py
To produce:

 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def cmd(path, *args, **kwargs):
"""
Generate a git log command using the the provided arguments. Empty git log
lines are stripped by this function.

Params:
[in] path - location of repo
[in] args - non-keyword arguments to provide to the git log command
[in] kwargs - keyword arguments to provide to the git log command

Returns: a list of lines returned from the git log command
"""
repo = Repo(path)
assert repo.bare == False
result = list()
for line in repo.git.log(*args, **kwargs).split('\n'):
if len(line) == 0:
continue
result.append(line)
return result

The horizontal scrollbar shows up in the codeblock embedded in the file.

 The crude but effective source code to produce the same result:

 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from BeautifulSoup import BeautifulSoup as beautify
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
import sys

file = open(sys.argv[1], 'r')

print("""\
<style>
#codeblock {
overflow-x: scroll;
width: auto;
white-space: nowrap;
}
""")
print HtmlFormatter().get_style_defs('.highlight')
print("""\
</style>

<div id="codeblock">
""")
formatter = HtmlFormatter(linenos=True)
print highlight(file.read(), PythonLexer(), formatter)
print("""\
</div>
""")
file.close()
Update: GitHub Gists offers a script which greatly simplifies adding code to your blog.

July 19, 2017

Experiments with Vagrant and Packer On Fedora

  —A look at Vagrant and Packer (ancient Fedora).

I've had a strange experience with Packer on Fedora 13 (yes, 13). This may prove interesting to anyone encountered the following error from Packer:
==> virtualbox-iso: Error waiting for SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password keyboard-interactive], no supported methods remain
I'm using Packer v0.10.1.

I found that adjusting ssh_timeout did not seem to have any effect. I ended up setting ssh_handshake_attempts equal to 100.

What I learned in investigating this problem is this is the error message you get when your handshake attempts threshold exceeded. This message implies an authentication problem, but in my case it occurred because SSH wasn't running yet.

To remedy this, confirm whether it's an SSH problem or not by executing:
PACKER_LOG=1 PACKER_LOG_PATH=out.log packer build --debug TEMPLATE
Stop at the step:
==> virtualbox-iso: Pausing after run of step 'StepTypeBootCommand'. Press enter to continue.
Run Packer on a template with headless set to false (the default) so you can watch the virtual machine boot. Login once things are up and running. If the install completes you can also "Press enter to continue". If Packer connects over SSH you can rule out an SSH problem.

To confirm whether it's problem with your Packer template check ssh_wait_timeout and ssh_timeoutssh_wait_timeout is deprecated in favour of ssh_timeout.  Curiously, neither seemed to have any effect on Fedora 13. At one point, I set my timeout to hours and watched Packer shutdown the virtual machine because it couldn't connect is the space of a few minutes.

I was successful using both parameters on CentOS 7.0 and 7.2 as well as Debian 7.11.0 and 8.5.0. 

The problem doesn't appear to be my Packer template (Fedora uses the same template as CentOS and Debian). 

It isn't Packer (it works properly on CentOS and Debian).

Weird.

July 13, 2017

A Poor Use of GitPython

  —Introducing dependencies isn't always beneficial.

I'm working on a project that uses Git. It's written in Python. The initial implementation uses GitPython. GitPython provides abstractions for using Python with Git repositories.

My project uses git-log. I chose a layered approach using this function as a single point of access to the Git repository.

This function is the focal point for access to the git repository.
 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
def cmd(path, *args, **kwargs):
"""
Generate a git log command using the the provided arguments. Empty git log
lines are stripped by this function.

Params:
[in] path - location of repo
[in] args - non-keyword arguments to provide to the git log command
[in] kwargs - keyword arguments to provide to the git log command

Returns: a list of lines returned from the git log command
"""
repo = Repo(path)
assert repo.bare == False
result = list()
for line in repo.git.log(*args, **kwargs).split('\n'):
if len(line) == 0:
continue
result.append(line)
return result

Layering separates the repository interface from the information in the repository. It isolates the GitPython interface from the rest of the project but introduces a different challenge.

Consider this function.
 1
2
3
4
5
6
7
8
9
10
11
def commitLog(path, commitHash):
"""
Return the log entry for a single commit hash.

Params:
[in] path - location of repo
[in] commitHash - commit hash

Returns: log entry for the specified commit hash
"""
return ' '.join(cmd(path, '-n 1', commitHash)[3:])

My approach to layering moves the command-line options for git-log into the clients of cmd().

A better approach might have used GitPython's object model. Unfortunately the learning curve requires knowledge of git internals. Something I wanted to avoid in the interest of time. In my haste I didn't take advantage of GitPython's power.

The next revision of this project needs to revisit GitPython's object model to help traverse my repository. It will be interesting to see if this model supports obtaining information  provided by git log -n 1 and other porcelain options easily.

June 20, 2017

Experiments with Vagrant and Packer on Debian

  —A look at Vagrant and Packer (Debian).

I had occasion recently to review Vagrant and Packer from Hashi Corp. My review involved an exploration of whether Packer is useful for bootstrapping virtual machine. I ended up selecting a project by Pierre Mavro called packer-debian.

I am impressed with the power provided by Packer and Vagrant. They achieve the goal of automating the construction and deployment of virtual machines while enabling control of the virtual machine configuration. Very powerful tools.

My only disappointment with Packer is that the JSON files used to configure the virtual machine do not seem to support comments.

My experiments are located on GitHub. Ultimately, my experiments are directed at creating virtual machines for ancient operating systems. My immediate goal is to ensure that I can build an application on Fedora 13. (Yes, 13. Long story.). I will publish my work on GitHub as it progresses.

With thanks to Pierre Mavro for publishing his work. It enabled me to use a working example as a basis for my own investigation.