Merge branch 'main' of github.com:VariantSync/DiffDetective into main

Author: pmbittner

.gitattributes

@@ -0,0 +1,11 @@
+# Set the default behavior, in case people don't have core.autocrlf set.
+* text=auto
+
+# Explicitly declare text files you want to always be normalized and converted
+# to native line endings on checkout.
+*.sh text eol=lf
+*.bat text eol=crlf
+
+# Denote all files that are truly binary and should not be modified.
+*.png binary
+*.jpg binary

.gitignore

@@ -121,12 +121,11 @@ patches/*
 /input
 /error
 examples/
-results/
 
 /log.txt
-/local-maven-repo
 
 ### Eclipse ###
 .settings/
 .classpath
 .project
+plotting/__pycache__

Dockerfile

@@ -0,0 +1,75 @@
+# syntax=docker/dockerfile:1
+# ----------------------------
+# This template sets up a Docker container using a multi-stage build and Alpine Linux.
+# Alpine is a Linux-based OS embedded systems making it one of the smalles Docker images. It should be used if possible.
+# The multi-stage build consists of two stages: The compile stage, and the environment preparation stage.
+#
+# The compile stage only installs the packages required to compile the source files of the prototype. The generated binaries
+# are then copied to the environment during the environment preparation stage.
+#
+# The environment preparation stage is responsible for installing all dependencies and copying all files that are required
+# to execute the Docker container.
+#
+# This template contains the commands that are required to compile and execute a Java prototype.
+#
+# Lines commented as `REQUIRED` should only be altered/removed if you know what you are doing.
+# Lines commented as `EXAMPLE` contain commands that probably have to be adjusted
+# ----------------------------
+
+
+FROM alpine:3.15
+# PACKAGE STAGE
+
+# EXAMPLE: Prepare the compile environment. JDK is automatically installed
+RUN apk add maven
+
+# REQUIRED: Create and navigate to a working directory
+WORKDIR /home/user
+
+COPY local-maven-repo ./local-maven-repo
+
+# EXAMPLE: Copy the source code
+COPY src ./src
+# EXAMPLE: Copy the pom.xml if Maven is used
+COPY pom.xml .
+# EXAMPLE: Execute the maven package process
+RUN mvn package || exit
+
+FROM alpine:3.15
+
+# Create a user
+RUN adduser --disabled-password  --home /home/sherlock --gecos '' sherlock
+
+RUN apk add --no-cache --upgrade bash
+RUN apk add --update openjdk17
+
+# REQUIRED: Change into the home directory
+WORKDIR /home/sherlock
+
+# Copy the compiled JAR file from the first stage into the second stage
+# Syntax: COPY --from=STAGE_ID SOURCE_PATH TARGET_PATH
+WORKDIR /home/sherlock/holmes
+COPY --from=0 /home/user/target/DiffDetectiveRunner.jar .
+WORKDIR /home/sherlock
+
+# Copy the setup
+COPY docs holmes/docs
+
+# Copy the docker resources
+COPY docker/* ./
+RUN mkdir DiffDetectiveMining
+
+#  Adjust permissions
+RUN chown sherlock:sherlock /home/sherlock -R
+RUN chmod +x execute.sh
+RUN chmod +x entrypoint.sh
+RUN chmod +x fix-perms.sh
+
+# EXAMPLE: List the content in the work dir
+RUN ls -l
+
+# REQUIRED: Set the entrypoint
+ENTRYPOINT ["./entrypoint.sh", "./execute.sh"]
+
+# REQUIRED: Set the user
+USER sherlock

INSTALL.md

@@ -0,0 +1,118 @@
+# Installation
+## Installation Instructions
+In the following, we describe how to build the Docker image and run the experiments in Docker containers.
+
+### Install Docker (if required)
+How to install Docker depends on your operating system.
+
+#### Windows or Mac
+You can find download and installation instructions [here](https://www.docker.com/get-started).
+
+#### Linux Distributions
+How to install Docker on your system, depends on your distribution. However, the chances are high that Docker is part of your distributions package database.
+Docker's [documentation](https://docs.docker.com/engine/install/) contains instructions for common distributions.
+
+### Open a Suitable Terminal
+```
+# Windows Command Prompt: 
+ - Press 'Windows Key + R' on your keyboard
+ - Type in 'cmd' 
+ - Click 'OK' or press 'Enter' on your keyboard
+ 
+# Windows PowerShell:
+ - Open the search bar (Default: 'Windows Key') and search for 'PowerShell'
+ - Start the PowerShell
+ 
+# Linux:
+ - Press 'ctrl + alt + T' on your keyboard
+```
+
+### Build the Docker Container
+To build the Docker container you can run the build script corresponding to your OS
+```
+# Windows: 
+  .\build.bat
+# Linux/Mac (bash): 
+  ./build.sh
+```
+
+## Validation & Expected Output
+
+### Running the Validation
+To run the validation you can run the script corresponding to your OS with `validation` as first argument. The validation should take about 10-20 minutes depending on your hardware.
+```
+# Windows: 
+  .\execute.bat validation
+# Linux/Mac (bash): 
+  ./execute.sh validation
+```
+The results of the validation will be stored in the [results](results) directory.
+
+### Expected Output of the Validation
+The aggregated results of the validation can be found in the following files.
+
+- The [speed statistics](results/difftrees/speedstatistics.txt) contain information about the total runtime, median runtime, mean runtime, and more:
+  ```
+  #Commits: 14527
+  Total   commit process time is: 12.427866666666667min
+  Fastest commit process time is: df4a1fa9c5cc5d54a9347a2bf4843cae87a942f1___xorg-server___0ms
+  Slowest commit process time is: 9838b7032ea9792bec21af424c53c07078636d21___xorg-server___14578ms
+  Median  commit process time is: 6dc71f6b2c7ff49adb504426b4cd206e4745e1e3___xorg-server___19ms
+  Average commit process time is: 51.330075032697735ms
+  ```
+- The [classification results](results/difftrees/ultimateresult.metadata.txt) contain information about how often each pattern was found, and more.
+  ```
+  repository: <NONE>
+  total commits: 18046
+  filtered commits: 593
+  failed commits: 0
+  empty commits: 2926
+  processed commits: 14527
+  tree diffs: 55008
+  fastestCommit: df4a1fa9c5cc5d54a9347a2bf4843cae87a942f1___xorg-server___0ms
+  slowestCommit: 9838b7032ea9792bec21af424c53c07078636d21___xorg-server___14578ms
+  runtime in seconds: 747.5400000000001
+  runtime with multithreading in seconds: 137.22
+  treeformat: diff.difftree.serialize.treeformat.CommitDiffDiffTreeLabelFormat
+  nodeformat: mining.formats.ReleaseMiningDiffNodeFormat
+  edgeformat: mining.formats.DirectedEdgeLabelFormat with mining.formats.ReleaseMiningDiffNodeFormat
+  analysis: mining.strategies.PatternValidation
+  #NON nodes: 0
+  #ADD nodes: 0
+  #REM nodes: 0
+  filtered because not (is not empty): 132
+  AddToPC: { total = 260536; commits = 12703 }
+  AddWithMapping: { total = 27720; commits = 1447 }
+  RemFromPC: { total = 235017; commits = 11830 }
+  RemWithMapping: { total = 15381; commits = 1361 }
+  Specialization: { total = 4662; commits = 624 }
+  Generalization: { total = 7397; commits = 564 }
+  Reconfiguration: { total = 2231; commits = 258 }
+  Refactoring: { total = 5769; commits = 921 }
+  Untouched: { total = 0; commits = 0 }
+  #Error[#else after #else]: 2
+  #Error[#endif without #if]: 8
+  #Error[#else or #elif without #if]: 9
+  #Error[not all annotations closed]: 6
+  ```
+  
+(Note that the above links only have a target after running the validation.)
+The processing times might deviate.
+
+## Troubleshooting
+
+### 'Got permission denied while trying to connect to the Docker daemon socket'
+`Problem:` This is a common problem under Linux, if the user trying to execute Docker commands does not have the permissions to do so. 
+
+`Fix:` You can fix this problem by either following the [post-installation instructions](https://docs.docker.com/engine/install/linux-postinstall/), or by executing the scripts in the replication package with elevated permissions (i.e., `sudo`)
+
+### 'Unable to find image 'replication-package:latest' locally'
+`Problem:` The Docker container could not be found. This either means that the name of the container that was built does not fit the name of the container that is being executed (this only happens if you changed the provided scripts), or that the Docker container was not built yet. 
+
+`Fix:` Follow the instructions described above in the section `Build the Docker Container`.
+
+### No results after validation, or 'cannot create directory '../results/difftrees': Permission denied'
+`Problem:` This problem can occur due to how permissions are managed inside the Docker container. More specifically, it will appear, if Docker is executed with elevated permissions (i.e., `sudo`) and if there is no [results](results) directory because it was deleted manually. In this case, Docker will create the directory with elevated permissions, and the Docker user has no permissions to access the directory.
+
+`Fix:` If there is a _results_ directory delete it with elevated permission (e.g., `sudo rm -r results`). 
+Then, create a new _results_ directory without elevated permissions, or execute `git restore .` to restore the deleted directory.

README.md

@@ -1,42 +1,66 @@
-# DiffDetective
-
-[![Thesis](https://img.shields.io/badge/Thesis-Read-blue)][thesis]
-
-This is the tool accompanying the bachelor's thesis [**Empirical Evaluation of Feature Trace Recording on the Edit History of Marlin**][thesis] by Sören Viegener.
-(The version of DiffDetective described in and submitted with the thesis can be found on branch `thesis-sv`).
-
-DiffDetective is a library to analyse the evolution of variability in source code in preprocessor-based software product lines.
-It serves two main purposes:
-1. DiffDetective parses diffs on preprocessor annotated source code to so called `DiffTrees`. For example, the following diff in which
-   the annotation `DEBUG` gets removed and an `else` case is added
-    ```diff
-      #if A
-        x = 0;
-    -   #if DEBUG
-          print(x);
-    -   #endif
-    + #else
-    +   x = 1;
-      #endif
-    ```
-   can be parsed to the following graph structure to analyse the diff:
-
-    ![difftreeshowcase](docs/showcase/examplediff.png)
-
-3. DiffDetective takes a preprocessor-based software product line repository as input and matches edit patterns in its commit history.
-It can detect an extensible variety of different edit patterns and reverse engineers feature contexts known from feature trace recording.
-The output of the tool consists of all pattern matches found and different metrics relevant for the evaluation of feature trace recording.
-
-## Related Work
-
-**Feature Trace Recording**.
-Paul Maximilian Bittner, Alexander Schultheiß, Thomas Thüm, Timo Kehrer, Jeffrey Young, and Lukas Linsbauer.
-*ESEC/FSE'21. ACM, New York, NY, USA. August 2021*:
-https://pmbittner.github.io/FeatureTraceRecording/
-
-### Edit Patterns
-**Concepts, Operations, and Feasibility of a Projection-Based Variation Control System**.
-Stefan Stănciulescu, Thorsten Berger, Eric Walkingshaw, and Andrzej Wąsowski.
-https://ieeexplore.ieee.org/document/7816478
-
-[thesis]: https://oparu.uni-ulm.de/xmlui/handle/123456789/38679
+# Classifying Edits to Variability in Source Code
+
+This is the replication package our submission _Classifying Edits to Variability in Source Code_ submitted to the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) in March 2022.
+
+This replication package consists of four parts:
+
+1. **Appendix**: The appendix of our paper is given in PDF format in the file [appendix.pdf](appendix.pdf).
+2. **DiffDetective**: For our validation, we built DiffDetective, a java library and command-line tool to classify edits to variability in git histories of preprocessor-based software product lines.
+3. **Haskell Formalization**: We provide an extended formalization in the Haskell programming language as described in our appendix. Its implementation can be found in the Haskell project in the [proofs](proofs) directory.
+4. **Dataset Overview**: We provide an overview of the 44 inspected datasets with updated links to their repositories in the file [docs/datasets.md](docs/datasets.md).
+
+## Appendix
+
+Our appendix consists of:
+1. An extended formalization of our concepts in the [Haskell][haskell] programming language. The corresponding source code is also part of this replication package (see below).
+2. The proofs for (a) the completeness of variation tree diffs to represent edits to variation trees, and (b) the completeness and unambiguity of our elementary edit patterns.
+3. An inspection of edit patterns from related work to show that existing patterns are either composite patterns built from our elementary patterns or similar to our elementary patterns.
+4. The complete results of our validation for all 44 datasets.
+
+## DiffDetective
+We offer a [Docker](https://www.docker.com/) setup to easily __replicate__ our validation with _DiffDetective_. 
+You can find detailed information on how to install Docker and build the container in the [INSTALL](INSTALL.md) file.
+In the following, we provide instructions for running the replication.
+
+### 1. Build the Docker container
+To build the Docker container you can run the _build_ script corresponding to your OS.
+#### Windows: 
+`.\build.bat`
+#### Linux/Mac (bash): 
+`./build.sh`
+
+### 2. Start the replication
+To execute the replication you can run the _execute_ script corresponding to your OS with `replication` as first argument.
+
+> ! The replication will at least require several hours and might require up to a few days depending on your system.
+> Therefore, we offer a short validation (5-10 minutes) which runs _DiffDetective_ on only four of the datasets.
+> You can run it by providing "validation" as argument instead of "replication" (i.e., ./execute.sh validation).
+> If you want to stop the replication, you can call the provided script for stopping the container. Note that you will have to restart the entire replication, if you stop it at any point.
+> #### Windows:
+> `.\stop-execution.bat`
+> #### Linux/Mac (bash):
+> `./stop-execution.sh`
+
+#### Windows: 
+`.\execute.bat replication`
+#### Linux/Mac (bash): 
+`./execute.sh replication`
+
+
+
+### 3. View the results in the [results](results) directory
+All raw results are stored in the [results](results) directory. The aggregated results can be found in the following files:
+- [speed statistics](results/difftrees/speedstatistics.txt): contains information about the total runtime, median runtime, mean runtime, and more.
+- [classification results](results/difftrees/ultimateresult.metadata.txt): contains information about how often each pattern was found, and more.
+
+(Note that the above links only have a target _after_ running the replication.)
+
+Moreover, the results comprise the (LaTeX) tables that are part of our paper and appendix.
+
+## Haskell Formalization
+The extended formalization in Haskell is a library using the _Stack_ build system.
+
+Instructions for manually installing Stack are given in [proofs/REQUIREMENTS.md](proofs/REQUIREMENTS.md).
+How to build our library and how to run the example is described in the [proofs/INSTALL.md](proofs/INSTALL.md).
+
+[haskell]: https://www.haskell.org/

appendix.pdf

@@ -0,0 +1,2 @@
+docker build -t replication-package .
+@pause

build.bat

@@ -0,0 +1,2 @@
+#! /bin/bash
+docker build -t replication-package .

build.sh

@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# FeatureIDE
+mvn deploy:deploy-file -DgroupId=de.ovgu -DartifactId=featureide.lib.fm -Dversion=3.8.1 -Durl=file:./local-maven-repo/ -DrepositoryId=local-maven-repo -DupdateReleaseInfo=true -Dfile=./lib/de.ovgu.featureide.lib.fm-v3.8.1.jar
+
+# Functjonal
+mvn deploy:deploy-file -DgroupId=anonymized -DartifactId=Functjonal -Dversion=1.0-SNAPSHOT -Durl=file:./local-maven-repo/ -DrepositoryId=local-maven-repo -DupdateReleaseInfo=true -Dfile=./lib/Functjonal-1.0-SNAPSHOT.jar
+
+# sat4j
+mvn deploy:deploy-file -DgroupId=org.sat4j -DartifactId=core -Dversion=2.3.5 -Durl=file:./local-maven-repo/ -DrepositoryId=local-maven-repo -DupdateReleaseInfo=true -Dfile=./lib/org.sat4j.core.jar

deploy_libs.sh

@@ -0,0 +1,13 @@
+# Docker Files
+
+This directory contains the files that are required to run the Docker container.
+
+## Permission Fix
+To fix permission issues that occur in Docker environments under Linux, two files are required: [`fix-perms.sh`](fix-perms.sh) and [`entypoint.sh`](entrypoint.sh).
+
+These files should remain unaltered and are automatically copied to the Docker container.
+
+> Make sure to also set the required [.gitattributes](../.gitattributes) in your replication package. They are required to assure that the Docker container can be executed correctly under Windows.
+
+## Execution
+The [`execute.sh`](execute.sh) script can be adjusted to run the program that should be executed by the Docker container.