Phill Conrad

Phill Conrad

Senior Lecturer, Computer Science

Posts by Collection

portfolio

publications

Partial order transport service for multimedia and other applications

P. D. Amer, C. Chassot, T. J. Connolly, M. Diaz and P. Conrad, "Partial-order transport service for multimedia and other applications," in IEEE/ACM Transactions on Networking, vol. 2, no. 5, pp. 440-456, Oct. 1994.

Investigates a partial-order connection (POC) service/protocol. Unlike classic transport services that deliver objects either in the exact order transmitted or according to no particular order, POC provides a partial-order service, i.e. a service that requires some, but not all objects to be received in the order transmitted.

 Download PDF    DOI: 10.1109/90.336326

Retransmission-based partially reliable transport service: An analytic model

R. Marasli, P. D. Amer and P. T. Conrad, "Retransmission-based partially reliable transport service: an analytic model," Proceedings of IEEE INFOCOM '96. Conference on Computer Communications, San Francisco, CA, 1996, pp. 621-629 vol.2.

Many applications such as video and audio can tolerate loss. When the network layer provides a best-effort service such as on the Internet, the loss rate of the underlying network service may be higher than an application’s tolerance for loss. This paper analytically studies retransmission-based partially reliable transport (layer protocol) service.

 Download PDF    DOI: 10.1109/INFCOM.1996.493356

Multimedia document retrieval system using partially ordered/partially reliable transport service

Phillip T. Conrad, Edward Golden, Paul D. Amer, Rahmi Marasli, "Multimedia document retrieval system using partially ordered/partially reliable transport service," Proc. SPIE 2667, Multimedia Computing and Networking 1996, (25 March 1996);

We investigate the benefits of using a partially-ordered/partially-reliable (PO/PR) transport service for multimedia document retrieval over the Internet by implementing a prototype system. We introduce PMSL, a document specification language combining aspects of Little and Ghafoor’s OCPN with features for graceful degradation. We then show that for retrieval of PMSL documents, our PO/PR transport protocol (POCv2) provides several benefits over tradditional protocols such as TCP or UDP.

 Download PDF    DOI: 10.1117/12.235867

Optimizing partially ordered transport services for multimedia applications

R. Marasli, P. Amer, P. Conrad. Optimizing partially ordered transport services for multimedia applications. International Conference on Multimedia Modeling, 1996, pp. 185–204.

This paper investigates how much the selection of a linear extension affects system performance in a partially ordered transport service used for for multimedia applications. We first show how to identify better linear extensions of a partial order, and then determine the performance gains by using such linear extensions at the time of transmission.

 Download PDF    Link

Transport QoS over unreliable networks: no guarantees, no free lunch!

P. Conrad, P. Amer, E. Golden, S. Iren, R. Marasli, A. Caro. Transport QoS over unreliable networks: no guarantees, no free lunch! 5th IFIP Int'l Workshop on Quality of Service, Columbia Univ, 5/1997, pp. 315-318.

We describe an approach to transport QoS in unreliable networks that focuses on tradeoffs rather than guarantees. In particular, we investigate tradeoffs between qualitative QoS parameters such as order and reliability, and quantitative parameters such as delay and throughput.

 Download PDF    DOI: 10.1007/978-0-387-35170-4_35

An Analytic Study of Partially Ordered Transport Services

Rahmi Marasli, Paul D. Amer, Phillip T. Conrad, An analytic study of partially ordered transport services, Computer Networks and ISDN Systems, Volume 29, Issue 6, 1997, Pages 675-699, ISSN 0169-7552,

This paper presents an analytic model for investigating the throughput, delay and buffer utilization characteristics of partially ordered transport services. We analyze the effects of packet and ack losses as well as applications’ order requirements on overall system performance. The analytic model is verified by comparing its results against those of an OPNET simulation model.

 Download PDF    DOI: 10.1016/S0169-7552(96)00113-4

Partially reliable transport service

R. Marasli, P. D. Amer and P. T. Conrad, "Partially reliable transport service," Proceedings Second IEEE Symposium on Computer and Communications, Alexandria, Egypt, 1997, pp. 648-656.

An analytic model is presented for a partially reliable transport protocol based on retransmissions. The model illustrates tradeoffs between two QoS parameters (delay and throughput), and various levels of reliability. The model predicts that the use of a reliable transport service when an application only needs a partially reliable one causes considerable throughput decreases and delay increases in lossy networks. On the other hand, over lossy networks, an unreliable transport service is unable to respect an application’s loss tolerance. In lossy environments, partially reliable transport service avoids the extra cost of a reliable transport service, and, simultaneously, guarantees the minimal reliability that an application requires. A retransmission-based partially reliable transport service can be provided through either sender-based or receiver-based loss detection and recovery. Results show that both techniques provide almost identical reliability and delay. However, a sender-based approach provides better throughput than a receiver-based approach at high acknowledge loss rates.

 Download PDF    DOI: 10.1109/ISCC.1997.616080

Metrics for quantifying partially ordered transport services

R. Marasli, P. Amer, P. Conrad. Metrics for quantifying partially ordered transport services 6th Int'l Conf. on Telecommunication Systems, Nashville, 3/98

Partially ordered transport service offers a middle ground between ordered service and unordered service. For applications requiring only partial order rather than total order, partially ordered service provides performance improvements in terms of delay and buffer utilization. Intuitively, one expects a partially ordered service to provide relatively greater performance improvement when the partial order specied by the service user is more “flexible” (i.e., has fewer order constraints) and smaller performance improvement when the partial order is less “flexible” (i.e., has more order constraints). In this paper, we investigate this notion formally by proposing metrics for the “flexibility” of partial orders, and determining through an OPNET simulation how well these metrics correlate with expected performance.

 Download PDF    Link

Network-conscious compressed images over wireless networks.

Iren S., Amer P.D., Conrad P.T. (1998) Network-conscious compressed images over wireless networks. In: Plagemann T., Goebel V. (eds) Interactive Distributed Multimedia Systems and Telecommunication Services. IDMS 1998. Lecture Notes in Computer Science, vol 1483. Springer, Berlin, Heidelberg

We apply the concept of network-consciousness to image compression, an approach that does not simply optimize compression, but which optimizes overall performance when compressed images are transmitted over a lossy packet-switched network such as the Internet. Using an Application Level Framing philosophy, an image is compressed into path MTU-size Application Data Units (ADUs) at the application layer. Each ADU carries its semantics, that is, it contains enough information to be processed independently of all other ADUs. Therefore each ADU can be delivered to the receiving application out-of-order, thereby enabling faster progressive display of images. We explain why this approach is useful in general and specifically for wireless/heterogeneous environments.

 Download PDF    DOI: 10.1007/BFb0055313

Network-conscious compressed image transmission over battlefield networks

S. Iren, P. D. Amer, A. Caro, G. Sezen, M. Taube and P. T. Conrad, "Network-conscious compressed image transmission over battlefield networks," IEEE Military Communications Conference. Proceedings. MILCOM 98 (Cat. No.98CH36201), Boston, MA, USA, 1998, pp. 908-912 vol.3.

We introduce an image compression and transmission system for battlefield networks. The system is based on network-conscious image compression, an approach to compression that does not simply maximize compression, but which optimizes overall performance when compressed images are transmitted over a lossy, packet-switched battlefield network. Using an application level framing philosophy, an image is compressed into path-maximum transmission unit-size application data units (ADUs) at the application layer. Each ADU is independent of others and carries its own semantics, that is, each ADU is a self-contained unit possessing all information necessary for decoding and displaying the information within that packet. Each independent ADU can be delivered to the receiving application out-of-order, thereby enabling faster progressive display of the image. We combine network-conscious image compression with an embedded focusing feature to provide a system that can be used in battlefield scenarios such as telemedicine or intelligence gathering.

 Download PDF    DOI: 10.1109/MILCOM.1998.726967

Testing environment for innovative transport protocols

P. T. Conrad, P. D. Amer, M. Taube, G. Sezen, S. Iren and A. Caro, "Testing environment for innovative transport protocols," IEEE Military Communications Conference. Proceedings. MILCOM 98 (Cat. No.98CH36201), Boston, MA, USA, 1998, pp. 292-296 vol.1.

This paper describes the development of a test environment for innovative transport protocols. Central to this work is the development of a universal transport library (UTL). The UTL is a library of transport protocols that provides application programmers the ability to write to a single application programming interface (API), then test their application with many different transport protocols. The UTL also allows for rapid prototyping of transport protocols at the user level. The UTL has been incorporated into two multimedia communication systems designed to provide better performance over lossy networks by using innovative transport protocol features: NETCICATS (a Network-Conscious Image Compression and Transmission System) and ReMDoR (a Remote Multimedia Document Retrieval system). These three tools facilitate the evaluation of flexible transport protocols and compression techniques for multimedia communications over lossy battlefield networks.

 Download PDF    DOI: 10.1109/MILCOM.1998.722591

Network-conscious GIF image transmission over the Internet.

Paul D Amer, Sami Iren, Gul E Sezen, Phillip T Conrad, Mason Taube, Armando Caro, Network-conscious GIF image transmission over the Internet, Computer Networks, Volume 31, Issue 7, 1999, Pages 693-708, ISSN 1389-1286.

Traditional image compression techniques seek the smallest possible file size for a given level of image quality. By contrast, network-conscious image compression techniques take into account the fact that a compressed image will be transmitted over a packet-switched network that may lose and reorder packets. We describe GIFNCa, a network-conscious revision of the popular GIF89a standard. As with GIF89a, GIFNCa compresses an image using LZW encoding, however GIFNCa does so using an Application Level Framing approach. The data is segmented into path MTU-size data units, each of which can be independently decompressed and displayed on its own. Under lossy network conditions, when used in combination with an unordered transport protocol, GIFNCa permits faster progressive display at the receiver than GIF89a over an ordered transport protocol. This advantage comes in exchange for a small penalty in overall compression. This paper defines GIFNCa, and presents preliminary experimental data concerning this tradeoff. The overall goal of this research is to illustrate (1) the value of considering network characteristics in designing image formats, and (2) the value of unordered transport service.

 Download PDF    DOI: 10.1016/S0169-7552(98)00295-5

The transport layer: tutorial and survey

Sami Iren, Paul D. Amer, and Phillip T. Conrad. 1999. The transport layer: tutorial and survey. ACM Comput. Surv. 31, 4 (December 1999), 360-404.

Transport layer protocols provide for end-to-end communication between two or more hosts. This paper presents a tutorial on transport layer concepts and terminology, and a survey of transport layer services and protocols. The transport layer protocol TCP is used as a reference point, and compared and contrasted with nineteen other protocols designed over the past two decades. The service and protocol features of twelve of the most important protocols are summarized in both text and tables.

 Download PDF    DOI: 10.1145/344588.344609

Simple Reliable Multicast for Parallel Processing in Extended LANs

A typical problem for a parallel processing system involves broadcasting large amounts of data from a master to several worker programs. This paper describes a reliable-multicast method to reduce the communication costs of this distribution. Our solution relies on a tuple-space mechanism as implemented in the Synergy system. We present results showing that even a simple implementation of reliable multicast can dramatically improve performance

 Download PDF    DOI: 10.1109/LCN.2000.891080

ReMDoR: Remote Multimedia Document Retrieval over Partial Order Transport.

Phillip T. Conrad, Armando Caro, and Paul Amer. 2001. ReMDoR: remote multimedia document retrieval over partial order transport. In Proceedings of the ninth ACM international conference on Multimedia (MULTIMEDIA '01). ACM, New York, NY, USA, 169-180.

This paper presents results from performance experiments that demonstrate and quantify performance improvements when a PO/R transport service is used instead of an ordered/reliable service (O/R e.g., TCP) or an unordered/unreliable service (e.g. UDP). We first describe the Remote Multimedia Document Retrieval system (ReMDoR), an experimental application developed by the authors to evaluate the performance of remote document retrieval over a variety of transport protocols. We then provide a detailed analysis of experiments comparing O/R service to PO/R service for retrieval of a multimedia document. Our results show that between 5% and 10% loss, user-perceivable improvements in progressive display are obtained when PO/R service is used. These results suggest that when packet losses occur in an underlying packet-switched network, transport services providing reliable delivery over independent streams (such the emerging Internet protocol SCTP) are beneficial for retrieval of streaming multimedia.

 Download PDF    DOI: 10.1145/500141.500167

SCTP In Battlefield Networks

P. T. Conrad, G. J. Heinz, A. L. Caro, P. D. Amer and J. Fiore, "SCTP in battlefield networks," 2001 MILCOM Proceedings Communications for Network-Centric Operations: Creating the Information Force (Cat. No.01CH37277), McLean, VA, USA, 2001, pp. 289-295 vol.1.

The Stream Control Transmission Protocol (SCTP) is a new Internet standards track transport layer protocol SCTP was originally designed to transport PSTN signaling messages over IP networks, but is also capable of serving as a general purpose transport protocol. As such, SCTP provides an alternative that may be better able to satisfy the requirements of future battlefield networks than the traditional transport protocols, TCP and UDP. Unlike traditional transport protocols, SCTP allows multiple streams of messages within a single connection (or, in SCTP terminology, a single association). As the results in this paper show, this ability is particularly helpful in reducing latency for streaming multimedia in high loss environments. SCTP also provides features for multi-homing that may be helpful in high-mobility environments and additional security against denial-of-service attacks based on SYN flooding.

 Download PDF    DOI: 10.1109/MILCOM.2001.985806

Evaluation of Architectures for Reliable Server Pooling in Wired and Wireless Environments.

M. U. Uyar, Jianliang Zheng, M. A. Fecko, S. Samtani and P. T. Conrad, "Evaluation of architectures for reliable server pooling in wired and wireless environments," in IEEE Journal on Selected Areas in Communications, vol. 22, no. 1, pp. 164-175, Jan. 2004.

Reliable server pooling (RSP) allows a pool of redundant information sources to be viewed as a single transport endpoint and, therefore, it is able to provide persistent connections and balanced traffic for different applications. The Internet Engineering Task Force RSerPool Working Group has proposed an architecture to implement the RSP; it defines an overlay network providing an upper layer protocol or an application with a range of reliability services, from simple server selection to a fully automatic session-failover capability. The simulation experiments conducted in both wired and wireless environments show that the current version of the RSerPool works well in fixed and relatively reliable environments, but its performance worsens rapidly as the networks become more unreliable or mobile. The issues we have identified in wireless mobile ad hoc networks include network partition, high signaling overhead, difficulty in synchronization among name servers, and excessive aggressiveness in handling failures. Alternative design options for the RSP in wireless and mobile environments are introduced and evaluated.

 Download PDF    DOI: 10.1109/JSAC.2003.818806

eVoices: a website supporting outreach by attracting target groups to computer science through culturally relevant themes.

Sarah Jones, Alejandro Hernandez, Pablo Ortiz, Gerardo Aldana, Phillip Conrad, and Diana Franklin. 2010. eVoices: a website supporting outreach by attracting target groups to computer science through culturally relevant themes. J. Comput. Sci. Coll. 25, 4 (April 2010), 134-140.

A popular approach to introducing students to computer science is to engage middle-school students in fun programming activities. One challenge in such a program is attracting students who are not already positively predisposed to computing. In order to attract a wider audience, we designed and implemented a website that integrates culturally relevant activities to appeal to parents and children of target groups.

 Download PDF    Link

Animal tlatoque: attracting middle school students to computing through culturally-relevant themes

Diana Franklin, Phillip Conrad, Gerardo Aldana, and Sarah Hough. 2011. Animal tlatoque: attracting middle school students to computing through culturally-relevant themes. In Proceedings of the 42nd ACM technical symposium on Computer science education (SIGCSE '11). ACM, New York, NY, USA, 453-458.

A popular approach to introducing students to computer science is to involve middle-school students in engaging programming activities. One challenge in such a program is attracting students who are not already positively predisposed to computing. In order to attract a diverse audience, we developed a summer program based on culturally-relevant themes that appealed to our two target audiences, females and Latina/os. This paper describes our success in developing and implementing a computing curriculum and recruiting materials for a 2-week summer camp integrating two themes, animal conservation and Mayan culture. Scratch programming was used to engage students in creating animations about animals and Mayan culture, allowing them an interdisciplinary experience that combined programming, culture, biology, art, and storytelling. Our recruiting efforts resulted in an application pool that was 73% female and 67% Latina/o, with only 6.5% in neither group. We had 34 students complete the program. Pre- and post- surveys showed that the number of students citing computer science as their top choice for a career doubled and interest in computer science as a career more than tripled.

 Download PDF    DOI: 10.1145/1953163.1953295

Evaluating TCP-friendliness in light of Concurrent Multipath Transfer

Ilknur Aydin, Janardhan Iyengar, Phillip Conrad, Chien-Chung Shen, Paul Amer, Evaluating TCP-friendliness in light of Concurrent Multipath Transfer, Computer Networks, Volume 56, Issue 7, 2012, Pages 1876-1892, ISSN 1389-1286

In prior work, a CMT protocol using SCTP multihoming (termed SCTP-based CMT) was proposed and investigated for improving application throughput. SCTP-based CMT was studied in (bottleneck-independent) wired networking scenarios with ns-2 simulations. This paper studies the TCP-friendliness of CMT in the Internet. In this paper, we surveyed historical developments of the TCP-friendliness concept and argued that the original TCP-friendliness doctrine should be extended to incorporate multihoming and SCTP-based CMT.

 Download PDF    DOI: 10.1016/j.comnet.2012.01.010

Assessment of computer science learning in a scratch-based outreach program

Diana Franklin, Phillip Conrad, Bryce Boe, Katy Nilsen, Charlotte Hill, Michelle Len, Greg Dreschler, Gerardo Aldana, Paulo Almeida-Tanaka, Brynn Kiefer, Chelsea Laird, Felicia Lopez, Christine Pham, Jessica Suarez, and Robert Waite. 2013. Assessment of computer science learning in a scratch-based outreach program. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE '13). ACM, New York, NY, USA, 371-376.

Many institutions have created and deployed outreach programs for middle school students with the goal of increasing the number and diversity of students who later pursue careers in computer science. While these programs have been shown to increase interest in computer science, there has been less work on showing whether participants learn computer science content. We address two questions, one specific, and the other more general: (1) “What computer science did our middle school students learn in our interdisciplinary two-week summer camp?” (2) “How can computer science concepts be assessed in the context of Scratch-based outreach programs”? We address both questions by presenting the design of our summer camp, an overview of our curriculum, our assessment methodology, and our assessment results. Though the sample size is not statistically significant, the results show that a two-week, interdisciplinary, non-academic summer camp can be effective not only for engaging students, but also for imparting CS content. In just two weeks, with a curriculum not entirely focused on computer science, students displayed competence with event-driven programming, initialization of state, message passing, and say/sound synchronization. We have employed assessment methodologies that avoid written exams, an approach both outreach and classroom-based programs may find useful.

 Download PDF    DOI: 2445196.2445304

Hairball: Lint-inspired Static Analysis of Scratch Projects

Bryce Boe, Charlotte Hill, Michelle Len, Greg Dreschler, Phillip Conrad, and Diana Franklin. 2013. Hairball: lint-inspired static analysis of scratch projects. In Proceeding of the 44th ACM technical symposium on Computer science education (SIGCSE '13). ACM, New York, NY, USA, 215-220.

Scratch programming has risen in prominence, not only as a potential language for K-12 computer science, but also in introductory college courses. Unfortunately, grading Scratch programs is time-consuming, requiring manual execution of each program. Automation of this process is greatly complicated by the very reason Scratch is an attractive introductory language—the projects are multimedia in nature, requiring eyes and ears to fully appreciate. We propose Hairball, an automated system that can be used both by a student to point out potential errors or unsafe practices, and by a grader to assist in inspecting the implementation of Scratch programs. Because automatic analysis will not be able to determine the sensory effect, Hairball focuses instead on the implementation, including safe/robust programming practices, providing a “lint-like” tool for Scratch.

 Download PDF    DOI: 2445196.2445265

Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming

Kyle Dewey, Phillip Conrad, Michelle Craig, and Elena Morozova. 2017. Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). ACM, New York, NY, USA, 317-322.

A good suite of test inputs is an indispensable tool both for manual and automated assessment of student submissions to programming assignments. Yet, without a way to evaluate our test suites, it is difficult to know how well we are doing, much less improve our practice. We present a technique for evaluating a hand-generated test suite by comparing its ability to find defects against that of a test suite generated automatically using Constraint Logic Programming (CLP). We describe our technique and present a case study using student submissions for an assignment from a second-year programming course. Our results show that a CLP-generated test suite was able to identify significant defects that the instructor-generated suite missed, despite having similar code coverage.

 Download PDF    DOI: 10.1145/3059009.3059051

Achievement Goals in CS1: Replication and Extension

Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. 2018. Achievement Goals in CS1: Replication and Extension. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (SIGCSE '18). ACM, New York, NY, USA, 687-692.

Replication research is rare in CS education. For this reason, it is often unclear to what extent our findings generalize beyond the context of their generation. The present paper is a replication and extension of Achievement Goal Theory research on CS1 students. Achievement goals are cognitive representations of desired competence (e.g., topic mastery, outperforming peers) in achievement settings, and can predict outcomes such as grades and interest. We study achievement goals and their effects on CS1 students at six institutions in four countries. Broad patterns are maintained—mastery goals are beneficial while appearance goals are not—but our data additionally admits fine-grained analyses that nuance these findings. In particular, students’ motivations for goal pursuit can clarify relationships between performance goals and outcomes.

 Download PDF    DOI: 10.1145/3159450.3159452

Listening to early career software developers

Michelle Craig, Phill Conrad, Dylan Lynch, Natasha Lee, and Laura Anthony. 2018. Listening to early career software developers. J. Comput. Sci. Coll. 33, 4 (April 2018), 138-149.

Previous work finds that recent college graduates entering the software development industry encounter difficulties early in their careers, due to significant differences between their coding experiences in academia vs. what is expected of them on the job. To explore the gap between academic and industry software development, we conducted interviews with twenty early career software developers with four-year degrees in CS. We present an analysis of these interviews, including excerpts in the developers’ own words, organized around six themes. We conclude with thoughts on how to bridge this gap so that our students may be better prepared when we launch them into their careers.

 Download PDF    Link

Undergraduate Teaching Assistants in Computer Science: A Systematic Literature Review

Diba Mirza, Phillip T. Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate Teaching Assistants in Computer Science: A Systematic Literature Review. In Proceedings of the 2019 ACM Conference on International Computing Education Research (ICER '19). Association for Computing Machinery, New York, NY, USA, 31–40. DOI:https://doi.org/10.1145/3291279.3339422

We present a systematic literature review of the prior work on Undergraduate Teaching Assistants (UTAs) in Computer Science with two goals: (1) to create a taxonomy of practices that relate to the design and implementation of UTA programs, (2) to identify the benefits of using UTAs as claimed by the literature and characterize the level of evidence for those claims. We analyze 336 excerpts from 40 papers related to these goals. We use content analysis on excerpts describing practices to extract high-level themes that include recruiting, UTA and program coordinator duties, training, evaluation and organization of UTA programs. We perform a more fine-grained analysis within each theme to identify specific questions about UTA programs and the answers provided by the literature. Using a similar technique, we report on the claimed benefits of UTA programs to students, UTAs, instructors and institutions. Our analysis follows well-defined protocols involving multiple reviewers and we report on the inter-rater reliability. The results provided in this paper lay the groundwork for developing evidence-based best practices in UTA programs and inform practice and policy related to the use of UTAs at tertiary institutions. As such, it is relevant to educators establishing a new UTA program, expanding an existing program, or continuously improving an established program, as well as those designing research studies of such programs.

 Download PDF    Link

Evaluating Commit, Issue and Product Quality in Team Software Development Projects

Christopher Hundhausen, Adam Carter, Phillip Conrad, Ahsun Tariq, and Olusola Adesope. 2021. Evaluating Commit, Issue and Product Quality in Team Software Development Projects. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, 108–114. DOI:https://doi.org/10.1145/3408877.3432362

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams’ processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.

 Download PDF    Link

Teaching Testing with Modern Technology Stacks in Undergraduate Software Engineering Courses

Scott P. Chow, Tanay Komarlu, and Phillip T. Conrad. 2021. Teaching Testing with Modern Technology Stacks in Undergraduate Software Engineering Courses. In 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 (ITiCSE 2021), June 26–July 1, 2021, Virtual Event, Germany. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3430665.3456352

Students’ experience with software testing in undergraduate computing courses is often relatively shallow, as compared to the importance of the topic. This experience report describes introducing industrial-strength testing into CMPSC 156, an upper division course in software engineering at UC Santa Barbara. We describe our efforts to modify our software engineering course to introduce rigorous test-coverage requirements into full-stack web development projects, requirements similar to those the authors had experienced in a professional software development setting. We present student feedback on the course and coverage metrics for the projects. We reflect on what about these changes worked (or didn’t), and provide suggestions for other instructors that would like to give their students a deeper experience with software testing in their software engineering courses.

 Download PDF    Link

Assessing individual contributions to software engineering projects: a replication study

C.D. Hundhausen, P.T. Conrad, A.S. Carter & O. Adesope (2022) Assessing individual contributions to software engineering projects: a replication study, Computer Science Education, 32:3, 335-354, DOI: 10.1080/08993408.2022.2071543

Background and Context: Assessing team members’ indivdiual contributions to software development projects poses a key problem for computing instructors. While instructors typically rely on subjective assessments, objective assessments could provide a more robust picture. To explore this possibility, In a 2020 paper, Buffardi presented a correlational analysis of objective metrics and subjective metrics in an advanced software engineering project course (n= 41 students and 10 teams), finding only two significant correlations. Objective: To explore the robustness of Buffardi’s findings and gain further insight, we conducted a larger scale replication of the Buffardi study (n = 118 students and 25 teams) in three courses at three institutions. Method: We collected the same data as in the Buffardi study and computed the same measures from those data. We replicated Buffardi’s exploratory, correlational and regression analyses of objective and subjective measures. Findings: While replicating four of Buffardi’s five significant correlational findings and partially replicating the findings of Buffardi’s regression analyses, our results go beyond those of Buffardi by identifying eight additional significant correlations. Implications: In contrast to Buffardi’s study, our larger scale study suggests that subjective and objective measures of individual performance in team software development projects can be fruitfully combined to provide consistent and complementary assessments of individual performance.

 Download PDF    Link

Investigating Reflection in Undergraduate Software Development Teams: An Analysis of Online Chat Transcripts

Christopher Hundhausen, Phill Conrad, Olusola Adesope, Ahsun Tariq, Samir Sbai, and Andrew Lu. 2023. Investigating Reflection in Undergraduate Software Development Teams: An Analysis of Online Chat Transcripts. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 743–749. https://doi.org/10.1145/3545945.3569790

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students’ use of reflection–one aspect of metacognition–in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams’ online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams’ ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams’ use of reflection in team software projects.

 Download PDF    Link

Combining GitHub, Chat, and Peer Evaluation Data to Assess Individual Contributions to Team Software Development Projects

Christopher Hundhausen, Phill Conrad, Olusola Adesope, and Ahsun Tariq. 2023. Combining GitHub, Chat, and Peer Evaluation Data to Assess Individual Contributions to Team Software Development Projects. ACM Trans. Comput. Educ. 23, 3, Article 33 (September 2023), 23 pages. https://doi.org/10.1145/3593592

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams (n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

 Download PDF    Link

An Empirical Study of the Content and Quality of Sprint Retrospectives in Undergraduate Team Software Projects

Christopher Hundhausen, Phill Conrad, Ahsun Tariq, Surya Pugal and Bryan Zamora Flores. 2024. An Empirical Study of the Content and Quality of Sprint Retrospectives in Undergraduate Team Software Projects. 46th International Conference on Software Engineering (ICSE 2024). Lisbon, Portugual. April 2024, 10 pages.

The retrospective, or retro, is a fundamental component of the Agile process, widely used in both software engineering courses and industry. In a retro, teams come together at the end of a sprint to reflect on their team’s performance. We conducted an empirical study to explore three research questions concerning retros in undergraduate team projects: (1) What do students reflect on? (2) What is the quality of their reflections? (3) How do teams’ retros vary in terms of content and quality? Our study analyzed a corpus of 963 statements documented in the retros of 32 undergraduate software teams (n=182 students) enrolled in four software engineering courses at two North American universities. A content analysis revealed that teams reflected most often on their work, communication, and collaboration practices. Nearly a third of teams’ reflections focused on their general work practices, while nearly half focused on specific areas of the software development lifecycle—most prominently, pull requests, issues, and coding/testing/debugging. An analysis of the quality of teams’ retro reflections showed that only 13% provided justification for a strategy to be stopped, continued, or started. An analysis of team-by-team results indicated significant differences in teams’ retro content and quality. We compare these results to previous studies of retros in academia and industry, and consider their implications for software engineering education.

 Download PDF   

service

Santa Barbara High School, Computer Science Academy

2019 to present

  • Member of Visioning Committee: Participated in Quarterly Meetings of board of local tech industry and academic leaders that advise on curriculum matters for Computer Science program at local public high school

talks

Why and How to Spend a Sabbatical in Industry

Published:

Panel presentation on the benefits of faculty taking on a short-term full-time role in industry during a sabaticcal or summer. I discuss my experience as an AppFolio Faculty Fellow in Summer 2014. Other presenters include:

Demystifying Language about Students’ Varied Identities

Published:

The goal of this special session is to help attendees who are committed to diversity and inclusion learn to talk about different dimensions of identity (e.g., race, class, gender, sex, sexuality, etc.). Includes lightning talks and participant discussion, led by:

  • Moderator: Cheryl A. Swanier, Claflin University. (Moderator)
  • Jason T. Black, Florida A&M University. Topic: “Discussing Privilege, Racism, and Implicit Bias—from the Eyes of Black Men”
  • Kamau Bobb, Georgia Tech. Topic: “Race Labels”
  • April Browne, Butte College. Topic: “Family Friendly”
  • Phillip T. Conrad, UC Santa Barbara. Topic: “Gender and Sexuality”
  • Colleen Lewis, Harvey Mudd College. Topic: “Ability and Ableism”
  • Sheila Tejada, University of Southern California. Topic: “Model Minority”

teaching