Adventures in Dev: 2010

Thursday, November 18, 2010

Mimicking Hudson when you must

Just a quick blog about a situation I found myself in because my current employer refuses to use Hudon for CI testing of PHP projects.

I was looking for a way to use phpunit to run tests and create a website that nicely displays the results. I thought it would be simple to create an ant script which would run the tests using the --log-junit feature to create an xml document that the junitreport ant task can use to produce a quick and dirty website that will display the test stats. It turns out that the xml that phpunit emits is not well formed junit xml, so the ant task would not parse it.

I could have used an xslt file and linux's xsltproc to translate the phpunit xml into what junitreport needs but I don't think this is robust solution into the future.

So instead I found this website:
http://clockwerx.blogspot.com/search/label/prototype.js

Here Daniel uses the phpunit json logging feature (--log-json) to emit json which is then used in a simple php file to display the test results. It also allows you to filter based on the status of each test.

I took his code and converted it to use jQuery (pulled down using google api so you don't need to download it yourself) and moved the filter div to the top so it doesn't cover test results. Also, I removed the links to raw output, agile, and recent pages.

Since we run the tests nightly (or when there is an update), we email a unique link to our developers. So the page is expecting a test (get) parameter which is used to locate the correct json file to display.

The url sent to other developers will look like this:
http://localhost/results.php?test=2010-11-16_1219

You will have to find icons and place them in an 'images' directory located in the same directory as this file.

Here it is.. hope it helps you when your company decides not to use a proven tool:

<?php

$file = dirname(__FILE__) . '/' . $_GET["test"]. '.js';
if (!file_exists($file)) {
die("No unit tests have been run?<br/>Nothing found in " . $file);
}
$json = file_get_contents($file);

//Ugh
$json = '[' . str_replace("}{", "},{", $json) . ']';

?>
<html>
<head>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js"></script>
</head>
<body>
<style>
table {
border: 1px solid rgb(240, 240, 240);
}

.error {
border: 1px solid orange;
}

.fail {
border: 1px solid red;
}

.pass {
border: 1px solid green;
}

.cell {
width: 200px !important;
text-align: center;
padding: 0.5em;
}
.cell img {
display: block;
margin: auto;
}

h4 {
font-weight: normal !important;
font-style: italic;
}
</style>
<script type="text/javascript">
var total_passes = 0;
var total_failures = 0;
var total_errors = 0;
var total_skipped = 0;
var total_incomplete= 0;

function renderSuiteStart(item) {
var h2 = document.createElement("H2");
h2.appendChild(document.createTextNode(item.suite));

var p = document.createElement("p");
p.appendChild(document.createTextNode(item.tests + " tests in suite"));

var div = document.createElement("div");
var table = document.createElement("table");

div.id = item.suite;
table.id = item.suite + "_table";
table.className = '';

div.appendChild(h2);
div.appendChild(p);
div.appendChild(table);

$('#main').append(div);
}

function renderTest(item) {
var h3 = document.createElement("H3");
var h4 = document.createElement("H4");
h3.appendChild(document.createTextNode(item.test));

var p = document.createElement("p");
p.appendChild(document.createTextNode(item.time + " tests in suite"));

var tr = document.createElement("TR");
var th = document.createElement("TH");
var message = document.createElement("TD");
var detail = document.createElement("TD");
var time = document.createElement("TD");
var img = document.createElement("IMG");
var backtrace = document.createElement("OL");

time.appendChild(document.createTextNode(item.time));

img.alt = item.status;
img.width="30";
img.height="30";
message.className = item.status + " cell";

switch (item.status) {
case 'error':
if (item.message == "Skipped Test") {
message.className = "skipped cell";
total_skipped++;
} else if (item.message == "Incomplete Test") {
message.className = "incomplete cell";
total_incomplete++;
} else {
total_errors++;
}
img.src = 'images/important.png';
break;
case 'fail':
total_failures++;
img.src = 'images/cross.png';
break;
default:
total_passes++;
img.src = 'images/check.png';
break;
}

$.each(item.trace,
function(i, t) {
var li = document.createElement("LI");
var text = t.file + "(line " + t.line + "): " + t.class + t.type + t.function + "()";
li.appendChild(document.createTextNode(text));

backtrace.appendChild(li);
}
);

h4.appendChild(document.createTextNode(item.test));
detail.appendChild(h4);
detail.appendChild(backtrace);

message.appendChild(img);
message.appendChild(document.createTextNode(item.message));
message.style.width = "300px";

tr.appendChild(message);
tr.appendChild(detail);

$("#"+ item.suite + "_table").append(tr);
}

function renderResults() {
var data = <?php print $json; ?>;

$.each(data, function(i,item) {

switch (item.event) {
case 'suiteStart':
renderSuiteStart(item);
break;
case 'test':
renderTest(item);
}
});
renderTOC();
}

function renderTOC() {
var menu = document.createElement('p');
var checkbox_pass = document.createElement('input');
var checkbox_fail = document.createElement('input');
var checkbox_error = document.createElement('input');
var checkbox_skipped = document.createElement('input');
var checkbox_incomplete = document.createElement('input');

var label_pass = document.createElement('label');
var label_fail = document.createElement('label');
var label_error = document.createElement('label');
var label_skipped = document.createElement('label');
var label_incomplete = document.createElement('label');
var status_image;

//menu.style.position = 'fixed';
menu.style.bottom = 0;
menu.style.right = "1.5em";
//menu.style.backgroundColor = 'rgb(240, 240, 240)';
menu.style.padding = "1em;";
menu.style.fontWeight = 'bolder';

checkbox_pass.type = "checkbox";
checkbox_fail.type = "checkbox";
checkbox_error.type = "checkbox";
checkbox_skipped.type = "checkbox";
checkbox_incomplete.type = "checkbox";

checkbox_pass.checked = "checked";
checkbox_fail.checked = "checked";
checkbox_error.checked = "checked";
checkbox_skipped.checked = "checked";
checkbox_incomplete.checked = "checked";

checkbox_pass.onclick=function(){toggle("pass")};
checkbox_fail.onclick=function(){toggle("fail")};
checkbox_error.onclick=function(){toggle("error")};
checkbox_skipped.onclick=function(){toggle("skipped")};
checkbox_incomplete.onclick=function(){toggle("incomplete")};

label_pass.appendChild(checkbox_pass);
label_fail.appendChild(checkbox_fail);
label_error.appendChild(checkbox_error);
label_skipped.appendChild(checkbox_skipped);
label_incomplete.appendChild(checkbox_incomplete);

label_pass.appendChild(document.createTextNode("Passed (" + total_passes + ")"));
label_fail.appendChild(document.createTextNode("Failed (" + total_failures + ")"));
label_error.appendChild(document.createTextNode("Error (" + total_errors + ")"));
label_skipped.appendChild(document.createTextNode("Skipped (" + total_skipped + ")"));
label_incomplete.appendChild(document.createTextNode("Incomplete (" + total_incomplete + ")"));

label_pass.style.color = "green";
label_fail.style.color = "red";
label_error.style.color = "#FF6600";
label_skipped.style.color = "blue";
label_incomplete.style.color = "gray";

status_image = document.createElement("IMG");
status_image.style.display = "block";
status_image.style.margin = "auto";
status_image.style.paddingright = "10px";
status_image.width="50";
status_image.height="50";
status_image.align="left";
if (total_failures > 0) {
status_image.src = "images/fail.gif";
status_image.title= "Some tests failed. Fix them and get a happy face";
} else {
status_image.src = "images/pass.jpg";
status_image.title= "All test passed!!! Give yourself a pat on the back, YOU ROCK!!";
}
menu.appendChild(status_image);

menu.appendChild(document.createElement("br"));
menu.appendChild(label_pass);
menu.appendChild(label_fail);
menu.appendChild(label_error);
menu.appendChild(label_skipped);
menu.appendChild(label_incomplete);

checkbox_pass.onclick();
checkbox_pass.checked = "";

checkbox_skipped.onclick();
checkbox_skipped.checked = "";

checkbox_incomplete.onclick();
checkbox_incomplete.checked = "";

$("#toc").append(menu);
}

function toggle(type) {

$("." + type).each(function(i, td) {
if (td.parentNode.style.display != 'none') {
td.parentNode.style.display = 'none';
} else {
td.parentNode.style.display = 'table-row';
}
});
}

$(document).ready(renderResults);
</script>
<div id="main" class="contentbox">
<h1>Unit tests</h1>
<p>
Generated: <?php print date("F j, Y, g:i a", filemtime($file)); ?> </p>
<div id="toc"/>
</div>
</body>
</html>

Thursday, October 21, 2010

I am looking for book ideas to refresh my machine learning knowledge. I have some but I wanted to see what the top universities are using these days.

Here are a couple of classes (Stanford has online courses):

Carnegie Mellon’s class:

Textbook: Pattern Recognition and Machine Learning , Chris Bishop.
Secondary textbook: The Elements of Statistical Learning: Data Mining, Inference, and Prediction Trevor Hastie, Robert Tibshirani, Jerome Friedman. 2nd edition.
Optional textbook: Machine Learning , Tom Mitchell.
Optional textbook: Information Theory, Inference, and Learning Algorithms , David Mackay.

MIT’s Class:

Cowell et al., "Probabilistic networks and expert systems", Springer-Verlag, 1999.
Bishop, "Neural Networks for Pattern Recognition", 1995
Duda, Hart, Stork, "Pattern Classification", 2000
Hastie, Tibshirani and Friedman, "Elements of Statistical Learning: Data Mining, Inference and Prediction", 2001
MacKay, "Information Theory, Inference, and Learning Algorithms", 2003. Available on-line here

Mitchell, "Machine Learning", 1997.
Cover and Thomas, "Elements of Information Theory", Wiley & Sons, 1991

Stanford's Class:
Very Interesting Video Course: http://academicearth.org/courses/machine-learning

There is no required text for this course. Notes will be posted periodically on the course web site. The following books are recommended as optional reading:

Christopher Bishop, Pattern Recognition and Machine Learning. Springer, 2006.
Richard Duda, Peter Hart and David Stork, Pattern Classification, 2nd ed. John Wiley & Sons, 2001.
Tom Mitchell, Machine Learning. McGraw-Hill, 1997.
Richard Sutton and Andrew Barto, Reinforcement Learning: An introduction. MIT Press, 1998

Course handouts and other materials can be downloaded from http://www.stanford.edu/class/cs229/materials.html

Let me know if you have any opinions about these books or any I missed.

Friday, May 14, 2010

Debugging PHP with Eclipse on Ubuntu

I have followed the instructions at the site below but there are many differences I found when I was setting up PHP debugging using XDebug with Eclipse on Ubuntu 10.04. So I wanted to take you through the steps that worked for me.

Older/incomplete instructions are found here:
http://www.phpeclipse.com/wiki/Howto/XDebugAndPHPEclipse

Program installation
First you will have to install the following (we use postgres and smarty so that may not be required for you):

eclipse, php5, php5-pgsql, apache2, smarty, zendframework, php5-xdebug

you can run this:

sudo apt-get install eclipse php5 php5-pgsql apache2 smarty zendframework php5-xdebug

In eclipse add a new update site in Help->Install New Software
Click the Add button and enter this site url:
http://update.phpeclipse.net/update/stable/1.2.x

Select the root check box to install all components of the PHPEclipse plugin

XDebug set up

Edit xdebug.ini:

sudo nano /etc/php5/apache2/conf.d/xdebug.ini

find a line that looks like this (or you may need to add one):
zend_extension=/usr/lib/php5/20090626+lfs/xdebug.so

Add the following lines for a local debug session:

xdebug.remote_enable=On
xdebug.remote_autostart=On
xdebug.remote_handler=dbgp
xdebug.remote_host=127.0.0.1
xdebug.remote_port=9000
xdebug.remote_mode=req

Apache Configuration

Edit apache2.conf settings:

sudo nano /etc/apache2/apache2.conf

Add an alias and directory for your project (this is only for testing):

Alias /project_name "/home/your user/workspace/project directory"
ServerName test.net
< Virtualhost *:80>
     DocumentRoot /home/your user/workspace
     ErrorLog /error.log
     CustomLog /access.log common
</ Virtualhost>

If you do not have ErrorLog in here, the error long is located:
/var/log/apache2/error.log
This is defined in another section in the apache2.conf file as well.

Restart Apache:

sudo /etc/init.d/apache2 restart

Set up Eclipse remote debug session:

Click on this link and follow the instructions in section XDebug Debug Profile

Set a break point by opening the file with the Eclipse php editor and right clicking in the left margin next to line you want to break on. Then select "Toggle XDebug Breakpoint"

To start the debug session go to:
http://localhost/your_prj_sub_directory/?XDEBUG_SESSION_START=ide_id_string

Navigate to the place in your site that would execute the code with the breakpoint and start debugging.

The local variables can take some time to show up after you go to that tab. And if you run into trouble (null pointer exception) with the custom expressions (i.e. watch expressions), then remove them. The PHPEclipse plugin does not seem to be very robust in the way it handles any exceptions your php expression may throw while you are stepping through.

Some times Eclipse (or the PHP plugin) gets a little flaky.
When this happens try:

refresh the page

"hard refresh" (ctr-F5)

restarting the PHP XDebug Remote Script

all else fails you will have to restart eclipse

For more configuration options here are some important file locations for Ubuntu 10.04:
php.ini: /etc/php5/apache2/php.ini
httpd.conf: /etc/apache2/httpd.conf

If there is anything that I missed please let me know and I will update these instructions.

Hope this help.