Navigating the Evolution of Large Language Models in Business Analysis: A Comparative Study of RAG, Prompt Engineering, and Fine-Tuning Techniques

Andrea Alberici – University of Tirana, Faculty of Economy, Rruga Arben Broci, 1, 1001, Tirana, Albania

Nevila Baci – University of Tirana, Faculty of Economy, Rruga Arben Broci, 1, 1001, Tirana, Albania

Abstract: The rapid advancements in large language models (LLMs) could prove to have significantly impacted the field of business analysis, particu­larly in the development of domain-specific languages (DSLs) tailored to de­scribe business requirements with precision and flexibility. The study high­lights the substantial progress in LLM capabilities, including extended con­text understanding, enhanced reasoning, and mathematical function­alities, which collectively facilitate deeper integration of domain-specific knowledge into business analysis processes.

The authors critically assess the relevance of Retrieval Augmented Gener­ative techniques that offer advanced knowledge injection methods, along with prompt engineering reasoning techniques, as opposed to fine-tun­ing LLMs. Furthermore, the research evaluates the strategic decision-mak­ing process for business analysts in adopting these technological advance­ments. The paper discusses whether business analysts should take a proac­tive or cautious approach when incorporating these AI-driven methodolo­gies into their analytical frameworks, or just wait for the next turn of LLM’s improvements.

By examining various case studies and conducting interviews with experts, this study provides insights into how the deliberate application of advanced LLM techniques can offset the services brought by RAG/Prompt engineer­ing techniques. The text also provides guidance for navigating the techno­logical landscape, indicating that it is important to stay updated with rap­id advancements. A strategic combination of RAG, prompt engineering, and fine-tuning can provide a balanced and effective approach to creating in­tentional frameworks that meet the evolving needs of businesses today.

8th International Scientific Conference – EMAN 2024 – Economics and Management: How to Cope With Disrupted Times, Rome, Italy, March 21, 2024, SELECTED PAPERS, published by: Association of Economists and Managers of the Balkans, Belgrade, Serbia; ISBN 978-86-80194-84-4, ISSN 2683-4510, DOI: https://doi.org/10.31410/EMAN.S.P.2024

Creative Commons Non Commercial CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution-Non-Commercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission. 

REFERENCES

AI Collective. (2024). https://www.ai-collective.co.uk/

Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Learning to Retrieve, Gener­ate, and Critique through Self-Reflection. https://doi.org/10.48550/arXiv.2310.11511

Bansal, L. (2024, January 16). A Complete Guide to RAG and LlamaIndex. Medium. https://pub.to­wardsai.net/a-complete-guide-to-rag-and-llamaindex-2e1776655bfa

Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A Pretrained Language Model for Scientific Text. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). https://doi.org/10.18653/v1/d19-1371

Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020). LE­GAL-BERT: The Muppets straight out of Law School. Findings of the Association for Compu­tational Linguistics: EMNLP 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.26

Cheng, D., Huang, S., & Wei, F. (2024). Adapting Large Language Models via Reading Comprehen­sion. https://doi.org/10.48550/arXiv.2309.09530

Cohere. (2024). Multi-step Tool Use (Agents). Cohere AI. https://docs.cohere.com/docs/multi-step-tool-use

Dell’Acqua, F., McFowland, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Kray­er, L., Candelon, F., & Lakhani, K. R. (2023a). Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Qual­ity. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4573321

Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Del Verme, M., Marty, T., Boisvert, L., Thakkar, M., Cappart, Q., Vazquez, D., Chapados, N., & Lacoste, A. (2024). WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? http://arxiv.org/abs/2403.07718

Eladlev. (2024). Eladlev/AutoPrompt. https://github.com/Eladlev/AutoPrompt

Fan, T., Kang, Y., Ma, G., Chen, W., Wei, W., Fan, L., & Yang, Q. (2023). Fate-llm: A industrial grade federated learning framework for large language models. arXiv preprint arXiv:2310.10049.

GuidanceAI. (2024). Guidance-ai/guidance. https://github.com/guidance-ai/guidance

Honchar, A. (2024, March 15). Intro to LLM Agents with Langchain: When RAG is Not Enough. Medium. https://towardsdatascience.com/intro-to-llm-agents-with-langchain-when-rag-is-not-enough-7d8c08145834

Hong, S., Lin, Y., Liu, B., Liu, B., Wu, B., Li, D., Chen, J., Zhang, J., Wang, J., Zhang, L., Zhang, L., Yang, M., Zhuge, M., Guo, T., Zhou, T., Tao, W., Wang, W., Tang, X., Lu, X., … Wu, C. (2024). Data Interpreter: An LLM Agent For Data Science https://doi.org/10.48550/arXiv.2402.18679

Hosni, Y. (2024, March 10). Prompt Engineering Best Practices: Iterative Prompt Development. https://medium.com/prompt-engineering-best-practices-iterative-prompt-development-22759b309919

Huang, C. (2024, February 28). Dpo_descriptiveness.py. Github. https://gist.github.com/vwxyzjn/64d91ce0b66b0548f1d2c33e855d168c

Jang, D.-H., Yun, S., & Han, D. (2024). Model Stock: All we need is just a few fine-tuned models. https://doi.org/10.48550/arXiv.2403.19522

Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. C. (2024). Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity https://doi.org/10.48550/arXiv.2403.14403

Khattab, O., Shandilya, H., & Singhvi, A. (2023). DSPy documentation. https://dspy-docs.vercel.app/docs/intro

Krishnamurthy, A., Harris, K., Foster, D. J., Zhang, C., & Slivkins, A. (2024). Can large language models explore in-context? http://arxiv.org/abs/2403.15371

Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Rozen, N., Schwartz, E., Zusman, M., & Shoham, Y. (2024). Jam­ba: A hybrid transformer-mamba language model. https://doi.org/10.48550/arXiv.2403.19887

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023). G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. https://doi.org/10.48550/arXiv.2303.16634

Nguyen, I. (2024, March). Evaluating RAG Part I: How to Evaluate Document Retrieval. https://www.deepset.ai/blog/rag-evaluation-retrieval

NousResearch. (2024). NousResearch/Genstruct-7B · Hugging Face. https://huggingface.co/NousResearch/Genstruct-7B

Olickel, H. (n.d.). Better RAG 1: Advanced Basics. https://huggingface.co/blog/hrishioa/retrieval-augmented-generation-1-basics

Olickel, H. (2024). Better RAG 2: Single-shot is not good enough. https://huggingface.co/blog/hrishioa/retrieval-augmented-generation-2-walking  

Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2024). MemGPT: Towards LLMs as operating systems. arXiv. https://arxiv.org/abs/2310.06177 

Sahota, H. (2023, October 20). RAG with LlamaIndex and DeciLM: A Step-by-Step Tutorial. Deci. https://deci.ai/blog/rag-with-llamaindex-and-decilm-a-step-by-step-tutorial/ 

Saravia, E. (2024, February 21). Removing RAG because of Context. X (Formerly Twitter). https://twitter.com/omarsar0/status/1760128830230978925  

Sher, D. V. (2024, February 16). Using LangChain ReAct Agents for Answering Mul­ti-hop Questions in RAG Systems. Medium. https://towardsdatascience.com/using-langchain-react-agents-for-answeringmulti-hop-questions-in-rag-systems-893208c1847e   

Tao, W., Zhou, Y., Zhang, W., & Cheng, Y. (2024). MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution. arXiv preprint arXiv:2403.17927. 

Wahle, J., Ruas, T., Abdalla, M., Gipp, B., & Mohammad, S. (2023). We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 12896-12913. https://doi.org/10.18653/v1/2023.emnlp-main.797 

Yang, C., Li, J., Niu, X., Du, X., Gao, S., Zhang, H., Chen, Z., Qu, X., Yuan, R., Li, Y., Liu, J., Huang, S. W., Yue, S., Chen, W., Fu, J., & Zhang, G. (2024). The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis https://doi. org/10.48550/arXiv.2404.01204 

Yang, C. J. (2024, March 24). Deterministic Document Structure based Retrieval. WhyHow.AI. https://medium.com/enterprise-rag/deterministic-document-structure-based-retrieval-472682f9629a  

Yi Tay. (2024, March 3). A New Open Source Flan 20B with UL2. https://www.yitay.net/blog/flan-ul2-20b 

Zhao, H., Liu, Z., Wu, Z., Li, Y., Yang, T., Shu, P., Xu, S., Dai, H., Zhao, L., Mai, G., Liu, N., & Liu, T. (2024). Revolutionizing Finance with LLMs: An Overview of Applications and Insights. https://doi.org/10.48550/arXiv.2401.11641