Beyond Patterns: Evaluating LLMs’ Capabilities in Detecting Secrets in Android Apps

In this repository, we host all artefacts related to our paper titled "Beyond Patterns: Evaluating LLMs’ Capabilities in Detecting Secrets in Android Apps".

📋 Setup/Requirements

This section provides instructions to set up the environment required to run LLM-based analysis.

🐍 Conda Environment

To launch the Jupyter Notebook, you will need various libraries. We provide a requirements.txt file which you can use to create a conda environment.

Follow the steps below:

1. Create a conda environment named demoEnv:

   conda create --name demoEnv python=3.11

2. Activate the newly created environment:

   conda activate demoEnv

3. Install the required packages using pip and requirements.txt:

   pip install -r requirements.txt

Once these steps are complete, your environment will be set up with all the necessary libraries.

🔧 ApkTool

To decompile APKs, ApkTool must be installed on your system. Follow the steps below to set it up:

1. Download ApkTool:
   Visit the official ApkTool page at https://ibotpeaches.github.io/Apktool/ and download the latest version.

2. Install ApkTool:
   Follow the installation instructions for your operating system, which typically involve:

   • Placing the downloaded JAR file in a suitable directory.
   • Adding the ApkTool executable to your system's PATH for easier access.

3. Verify Installation:
   Ensure ApkTool is installed correctly by running the following command in your terminal:

   apktool

   This should display the ApkTool usage instructions if the installation was successful.

🔑 Environment File (.env)

To execute the code, you will need to set up an environment file named .env in the main directory of the repository. This file contains all the necessary API keys and paths required for the experiments.

Below is the list of variables that should be included in the .env file:

• OPENAI_API_KEY – Your OpenAI API key (required for LLM-based analysis).
  Example: OPENAI_API_KEY=[YOUR_OPENAI_API_KEY]

If you prefer not to pay for OpenAI API usage, you can rely on a local Ollama server by specifying the OLLAMA_SERVER variable. The code will automatically detect and use it if provided.

• OLLAMA_SERVER – (Optional) URL of your local Ollama server, used if you prefer not to rely on paid OpenAI APIs.
  Example: OLLAMA_SERVER=http://localhost:11434

Other Variables:

• ANDROID_PATH – Absolute path to your Android SDK platforms directory, required by Soot to analyze APKs and extract strings.
  Example: ANDROID_PATH=/home/user/Android/Sdk/platforms

• ANDROZOO_API_KEY – Your API key to download applications from AndroZoo.
  You can request access from their website: https://androzoo.uni.lu/
  Example: ANDROZOO_API_KEY=[YOUR_ANDROZOO_API_KEY]

💸 Note: Please be aware that using OpenAI’s models may incur costs depending on the volume and type of API usage. Refer to OpenAI's pricing page for details.

---

⚙️ Usage

The experiments.ipynb Jupyter Notebook serves as the primary interface for conducting the LLM-based analysis.

To begin, place a CSV file in the 0_Data directory. Executing the notebook will then generate the corresponding results file.

Furthermore, the notebook enables users to specify the model employed in the analysis, either by selecting an OpenAI-provided model or by utilizing a model installed on a local Ollama server.

⚠️ Note:

Due to the probabilistic nature of LLMs, outputs may slightly vary between runs. As a result, you might observe small deviations in the generated results compared to those presented in the paper.

---

🔒 Dataset Access

Due to the sensitive nature of the data involved in this project (e.g., exploitable secrets such as API keys, access tokens, and other credentials) we cannot publicly release the full dataset analyzed in our study.

Researchers interested in accessing the complete dataset may contact us directly by email at ...

⚠️ Access will be granted only to authenticated researchers and exclusively for legitimate academic purposes.

🔍 Test Dataset for Reproducibility

To support reproducibility and transparency, we provide a safe test dataset consisting of apps for which all detected secrets have been revoked, and explicitly confirmed as non-active by the developers we contacted.

This test dataset can be freely used to run the notebook, validate the pipeline, and explore our analysis framework on real-world Android apps without exposing active or sensitive credentials.