LinkedIn and 3rd parties use essential and non-essential cookies to provide, secure, analyze and improve our Services, and to show you relevant ads (including professional and job ads) on and off LinkedIn. Learn more in our Cookie Policy.
In this newsletter, I'll share battle-tested insights and experiences from my recent journey building an MVP Gen AI Code Analysis product as well as my 20+ years in software development. If you're exploring or building Gen AI applications, my practical tips and lessons from the trenches could save you valuable time and resources.
Generative AI has revolutionized application development, offering powerful capabilities in text completion, summarization, translation, analysis and more. However, effectively leveraging these capabilities requires understanding some key principles.
When embarking on our Gen AI application, we began by integrating REST API calls to established services like OpenAI, Azure OpenAI, or Gemini APIs. While these well-documented APIs offer a straightforward entry point, the true complexity lies in orchestrating the surrounding ecosystem for seamless functionality and user value. Let's explore the hurdles we encountered and the strategies we employed to overcome them.
Setting up and maintaining the necessary infrastructure to handle large volumes of requests can be complex. For instance, you may need a robust backend which can process various simultaneous queries within a fraction of seconds and handles failures gracefully. In my experience, choosing a cloud provider like Azure or equivalent provides the necessary infrastructure services for both AI and other surrounding services required for productionizing an AI based application. Azure has a solid Open AI ecosystem that it offers as services and makes it easier to setup, manage and scale the application.
Safeguarding consumer data is paramount. Implement robust processes and algorithms to protect sensitive information, ensuring that customer data is never exposed in ways that could harm their business or violate their intellectual property rights. Establish and enforce a comprehensive privacy policy to maintain customer trust and comply with relevant regulations.
Designing systems that can scale to meet varying demands without performance degradation is vital. If your application experiences a sudden spike in traffic, it should be able to handle the load without slowing down or crashing. Open AI API offers Tiered subscriptions whereas Azure Open AI offers quotas and PTUs to handle various situations with Capacity. Understanding the Tiered subscriptions, Quotas , PTUs or similar offerings becomes critical in scaling the system for various volumes of Users.
Consider an e-commerce platform that uses Gen AI to provide product recommendations. While calling the API to get recommendations is simple, integrating these recommendations into the user interface, handling API rate limits, and ensuring data security requires careful planning and robust infrastructure. So when you start building an AI based application start putting plans around these aspects as well from the beging
Analyzing API costs on a per-request and per-user basis is crucial for shaping your pricing model. By meticulously tracking the resource consumption associated with each API call and user interaction, you gain valuable insights into the actual cost of service delivery. This granular understanding allows you to determine a fair and sustainable pricing structure that accurately reflects the value provided to each customer segment. Moreover, it empowers you to identify potential areas for optimization, such as optimizing API calls or adjusting resource allocation, ultimately enhancing cost efficiency and profitability while delivering a high-quality user experience.
The landscape of Large Language Models (LLMs) is vast and varied, with numerous models available, each boasting different strengths and weaknesses depending on the specific use case. Some excel at generating creative content, while others are adept at summarizing information or answering complex questions. To ensure optimal performance and results, thorough testing of LLM response quality is imperative. By evaluating different models against your specific requirements, you can identify the LLM that best aligns with your application's unique needs and delivers the most suitable and useful responses to your users.
I have covered this topic in detail in my previous newsletters. Effective prompt engineering is crucial for obtaining high-quality responses from Gen AI models. This is another critical area of knowledge for the team. This involves crafting clear, specific prompts and providing sufficient context to guide the model towards the desired output.
Specificity: Tailor the prompt to elicit specific types of responses. For example, if you want the model to summarize a document, provide clear instructions and context. Giving the AI relevant background information (like user history or relevant documents) helps it generate more informed and personalized responses.
Gen AI models process text in terms of tokens (words or subwords). Efficient management of these tokens is crucial for cost savings and performance optimization. Know your Input and Output token usage per request, per user and track it efficiently through your system. There are various Token calculation libraries available for that purpose like tiktoken, leverage them in your code. For example:
Ensure that the AI's capabilities are easily accessible, and create value to the users. This involves designing interfaces that are intuitive and user-friendly, allowing users to interact with the AI seamlessly. The application should guide the users through process of using the application efforlessly and the user should interact with it preferably with a few clicks and obtain great responses even if they are not good prompt engineers.
Consider an email drafting tool powered by Gen AI. The interface should allow users to easily input their requirements, such as the tone and purpose of the email, and provide clear options for generating, editing, and sending drafts. Tooltips, examples, and guided steps can enhance usability, ensuring users get the most out of the AI capabilities without frustration.
Handling and Validation: Properly handle and validate user inputs to ensure they are appropriate for the AI to process. For instance, if the AI is used to generate customer service responses, inputs should be validated to ensure they contain relevant information and are free from errors or malicious content. In our case we decided to design a system which doesn't require a user to write prompts. Internally we crafted our complex prompts and built an extensive prompt management system which makes it easy to create, edit and store prompts.
Decide whether to store AI-generated responses for future reference or analysis. This could be useful for applications that need to keep a history of interactions, such as customer support systems or code analysis applications. However, it also involves considerations around data privacy and security.
Example: In a content generation platform, user prompts (e.g., "Write a blog post about AI in healthcare") should be validated for completeness and relevance. The generated responses can be stored in a database for users to revisit, edit, or reuse later.
The Annual Computer Security Applications Conference (ACSAC) brings together leading researchers and practitioners, along with a diverse group of security professionals drawn from academia, industry, and government, gathered to present and discuss the latest cybersecurity results and topics. With peer reviewed technical papers, invited talks, panels, national interest discussions, and workshops, ACSAC continues its core mission of investigating practical solutions for computer and network security technologies.
This year, we will celebrate ACSAC's 40th edition. ACSAC's first edition was held in 1985 and is now one of the oldest computer security conferences, second only to the IEEE Symposium on Security and Privacy. We invite you to join us for this special anniversary!
As an internationally recognized forum where researchers, practitioners, and developers meet to learn and to exchange practical ideas and experiences in computer and network security, we invite you to submit your work. In addition to peer-reviewed papers on novel applied research, we also welcome case studies on real-world applications, panels featuring world experts, and workshops consisting of 1-2 day sessions on hot cybersecurity and privacy topics.
Students must first request and receive the OPT recommendation I-20 form from OIE before filing their OPT application to USCIS either via mail or electronically in USCIS's portal. Per 8 CFR 214.2(f)(11), failure to include the I-20 form with the OPT recommendation within the OPT application to USCIS will likely result in the student being denied the OPT benefit by USCIS. If you believe you filed the application with USCIS before first receiving the recommendation I-20 form from OIE, please join us during drop-in advising.
Optional Practical Training (OPT) is a benefit of the F-1 visa program that allows you to pursue experiential learning opportunities in the U.S. that directly relate to your program of study. While an OIE Designated School Official (DSO) recommends a student for OPT, it is the student who must apply for the work permit with the U.S. Citizenship and Immigration Service (USCIS). If the OPT benefit is approved, USCIS will issue an Employment Authorization Document (EAD). Students can utilize this benefit once per education level. There are three types of OPT: pre-completion OPT, post-completion OPT, and the 24-Month STEM OPT extension.
OIE offers OPT workshops throughout the year to help students understand the application process, status maintenance requirements, travel expectations, and more. For the schedule and join information of our upcoming virtual workshops, please click here. You will also find a pre-recorded workshop as well. You are not required to have any materials prepared for the workshop, but you are encouraged to review the required OPT guidance below before attending. Please join the event at least 5 minutes before the start time of the workshop as once the workshop begins, the advisor will close the workshop.