Generative AI and LLM interview questions and answers
By Vikram · Updated 26 Oct 2026
GenAI roles and AI hackathons expect you to understand what happens inside a large language model and how to build reliable apps on top of one. These are the questions that come up most, from the basics to system design.
How LLMs work
- LLM: a large neural network, usually a Transformer, trained to predict the next token on huge amounts of text, then tuned to follow instructions.
- Tokens: the chunks of text the model reads and writes, often parts of words. Context limits and pricing are counted in tokens.
- Self-attention: every token looks at every other token and weighs how relevant each is; cost grows with the square of the sequence length.
- Why divide by √dₖ in attention: to keep the dot products from growing with dimension and saturating the softmax.
- Positional encodings: attention ignores order on its own, so position information (sinusoidal, learned or rotary) is added.
- Temperature: scales randomness when sampling. Low is focused and repeatable; high is more varied.
- KV cache: stores the keys and values of earlier tokens so each new token is generated without recomputing the whole prefix.
Building with LLMs
- Embeddings: vectors where similar meanings are close together, compared with cosine similarity.
- RAG (retrieval-augmented generation): split documents into chunks, embed them, retrieve the most relevant chunks for each question and put them in the prompt. It gives the model fresh or private knowledge without retraining.
- Fine-tuning vs RAG: fine-tune to change behaviour, style or format; use RAG to add knowledge that changes.
- LoRA: freeze the base model and train small low-rank matrices added to its layers: far cheaper fine-tuning.
- RLHF: train a reward model on human preferences, then optimise the model against it with a penalty for drifting too far. DPO is a simpler alternative.
- Agents and tool use: the model chooses tools (search, code, APIs) step by step to complete a task.
Reliability and safety
- Hallucinations: fluent but false answers. Reduce them with grounding (RAG), citations and asking the model to say when it does not know.
- Prompt injection: instructions hidden in untrusted content (a web page, an email) that try to take over the model. Treat retrieved text as data and limit what tools can do.
- Evaluation: test sets of real questions, automatic checks, model-graded evaluation with spot checks, and tracking quality, latency and cost.
Tip: Building something is the fastest way to learn. Join an AI hackathon or a generative AI hackathon, and test your concepts with the AI questions with answers from the Daily Quiz.
Frequently asked questions
What is RAG in generative AI?
Retrieval-augmented generation: relevant documents are retrieved, usually with embeddings and vector search, and added to the prompt so the model answers from them. It keeps answers current and grounded without retraining.
What is the difference between fine-tuning and RAG?
Fine-tuning changes the model's weights to change its behaviour or style. RAG leaves the model unchanged and supplies knowledge at question time. Many apps use both.
What is a token in an LLM?
A unit of text the model processes, often a word piece. In English one token is roughly three quarters of a word on average.
What is temperature in an LLM?
A sampling setting that scales randomness: near 0 the model almost always picks the most likely token; higher values give more varied output.