ThesisLLMs4SE

Tools:
Search
Index
Webs

Impressum

Large Language Models in Software Engineering: A Critical Review and Future Research Directions

worked on by: Ali Bektas

Outline/Expose

The development of Transformer-based Large Language Models (LLMs) has led to a burgeoning interest in their applications within the Software Engineering (SE) domain, as evidenced by the surge in related publications. Existing surveys, such as those conducted by Zheng et al. (2023) and Hou et al. (2023), have documented the extensive utilization of LLMs in SE, exploring a variety of applications and methodologies. Fan et al. (2023) describe the field of LLMs for SE as rapidly developing but still in an embryonic stage, highlighting the significant potential and the necessity for ongoing academic exploration. Their survey specifically aims to identify challenges and open problems, presenting a critical analysis of the early stages of this emerging field. In light of the rapid development of this field, which complicates a comprehensive review, this thesis is aimed at further exploring the field of LLM-based Software Engineering. It focuses on identifying new open problems and challenges and delineating potential future directions to provide additional guidance for the community in this dynamically evolving field.

To guide this investigation, the study is framed around two research questions:

RQ1: How do open problems, applied methodologies, challenges, and limitations in the selected studies vary across different SE tasks?
RQ2: Given the task-specific and methodological contexts, how can the identified open problems, challenges, and limitations be addressed, and what potential directions can be proposed?

This research, through a detailed review of the literature identified by Hou et al. (2023), extracts, compares, and analyzes data to identify specific challenges and insights. It highlights data-related challenges in SE tasks related to planning and effort estimation, underlining the need for training datasets that adequately represent a diverse array of projects and programming languages. Additionally, it points out that while some research papers leverage the non-deterministic nature of LLMs in their evaluation strategies, many studies lack this consideration in their methodologies. In activities such as code generation, debugging, and bug fixing, several papers have recognized the trade-off between the advantages of language models like ChatGPT and the effort required to provide additional information for debugging purposes. There is ongoing research in this direction, indicating potential areas for extension.

The thesis employs a structured methodological framework comprising predefined attributes — Task Context, Evaluation Criteria, Methodological Approach, and Insights and Reflections. This framework facilitates a systematic comparative analysis aimed at elucidating the variety of open problems, challenges, and limitations in the practical application of LLMs across different SE tasks and methodologies. Through this analysis, the thesis contributes to a more nuanced understanding of these issues, highlighting significant areas where the field may develop and improve.

The aim of this research is to analyze and synthesize the broad spectrum of existing problems and limitations in the integration of LLMs into SE tasks, with a focus on deriving actionable insights and proposing potential research directions. These insights are intended to guide future studies, thereby improving the practical application of LLMs in SE and advancing the field with innovative solutions.

References:

[1] Zheng, Zibin et al. "Towards an understanding of large language models in software engineering tasks". arXiv preprint arXiv:2308.11396 (2023).

[2] Hou, Xinyi et al. "Large language models for software engineering: A systematic literature review". arXiv preprint arXiv:2308.10620 (2023).

[3] Fan, Angela et al. "Large language models for software engineering: Survey and open problems". arXiv preprint arXiv:2310.03533 (2023).

Current Preliminary Version:

Masterarbeit_Ali_Bektas__Systematisches_Literatur_Review__Eignung_von_ChatGPT_und_Co_als_Unterstützung_beim_Software_Engineering (6).pdf

Thesis Requirements

formulate requirements here (together with your adviser)