Krishnaram Kenthapadi's Homepage

Krishnaram Kenthapadi

Krishnaram Kenthapadi is the Chief Scientist of Fiddler AI, an enterprise startup building a responsible AI and ML monitoring platform. Previously, he was a Principal Scientist at Amazon AWS AI, where he led the fairness, explainability, privacy, and model understanding initiatives in Amazon AI platform. Prior to joining Amazon, he led similar efforts across different LinkedIn applications as part of the LinkedIn AI team, and served as LinkedIn's representative in Microsoft's AI and Ethics in Engineering and Research (AETHER) Advisory Board. He shaped the technical roadmap and led the privacy/modeling efforts for LinkedIn Salary product, and prior to that, served as the relevance lead for the LinkedIn Careers and Talent Solutions Relevance team, which powers search/recommendation products at the intersection of members, recruiters, and career opportunities. Previously, he was a Researcher at Microsoft Research Silicon Valley, where his work resulted in product impact (and Gold Star / Technology Transfer awards), and several publications/patents. He received his Ph.D. in Computer Science from Stanford University in 2006, under the supervision of Professor Rajeev Motwani. Before joining Stanford, he received his Bachelors degree in Computer Science and Engineering from Indian Institute of Technology-Madras.

Krishnaram's expertise is in the areas of fairness/transparency/explainability/privacy in AI/ML systems, algorithms for large datasets and graphs, data mining, web search, information retrieval, search and recommendation systems, social network analysis, and computational education. He has 20+ years of experience (including 15+ years in industry after his PhD), working on challenging problems in these areas, and has shaped the technical roadmap/design/development/launch of new AI products, provided technical vision, and steered company-wide initiatives in new domains such as MLOps/explainability/fairness/privacy, and improved business metrics for existing products via technology transfers. He has collaborated with over 70 people with diverse backgrounds and interests and mentored over 40 summer interns, resulting in 50+ publications, with 4500+ citations, and 150+ filed patents (69 granted) in his fields of interest. He serves regularly on the program committees of KDD, WWW, WSDM, and related conferences, and co-chaired the 2014 ACM Symposium on Computing for Development. His work has been recognized through awards at NAACL, WWW, SODA, CIKM, ICML AutoML workshop, and Microsoft's AI/ML conference (MLADS). He has presented lectures/tutorials on privacy, fairness, explainable AI, and responsible AI in industry at forums such as KDD '18 '19, WSDM '19, WWW '19 '20 '21, FAccT '20 '21, AAAI '20 '21, and ICML '21, and instructed a course on AI at Stanford.

Research

Fairness/Transparency/Explainability/Privacy in AI/ML systems, Algorithms for Large Datasets and Graphs, Data Mining, Web Search, Information Retrieval, Search and Recommendation Systems, Social Network Analysis, Computational Education.

Publications [Google Scholar] [Patents (69 Granted)] [Selected Projects]

Contact

Email:

kkenthapadi [at] gmail.com [LinkedIn] [Twitter]

Selected Projects:

Fairness, Explainability, Privacy, and Model Understanding Initiatives in Amazon AWS AI Platform: While at Amazon, Krishnaram led the fairness, explainability, privacy, and model understanding initiatives in Amazon AWS AI platform.
- Shaped the technical roadmap for Amazon SageMaker Clarify, a new capability of Amazon SageMaker that helps customers detect bias in machine learning (ML) models, and increase transparency by helping explain model behavior to stakeholders and customers (launched in December 2020; AWS Blog; Amazon Science Article [Amazon AI Fairness and Explainability Whitepaper] [Amazon SageMaker Clarify KDD'21 Paper] [Fairness Measures for Machine Learning Paper] [Overview Slides]) and guiding the roadmap for products such as Amazon SageMaker Clarify and Amazon SageMaker Debugger.
- Led scientific investigations on topics such as model tuning under fairness constraints (ICML'20 AutoML Workshop Best Paper Award), minimax group fairness, fairness in human-in-the-loop settings, fairness and explainability system design, robustness of explanations, debugging models using adversarial examples, ML debugger system design, ML model monitoring system design, and differentially private synthetic data generation.
- Co-organized Amazon-wide workshops/off-sites on fairness, on explainability, and on privacy in ML; Generated awareness on these topics through keynotes/talks at various internal forums.
- Provided technical guidance for the Amazon Research Award program.
- Presented keynotes at LinkedIn-MSR-IISc Workshop on Fairness in ML (slides) and EMNLP'20 PrivateNLP workshop (slides; video).
- Presented tutorials on explainable AI and on responsible AI in industry at AAAI '20 '21, FAccT '20 '21, WWW '20 '21 & ICML '21, combining experiences across Amazon, Google, LinkedIn & Microsoft.
- Co-organized ICLR'21 Workshop on Synthetic Data Generation.
Fairness, Transparency, Explainability, and Privacy Initiatives across LinkedIn Products: While at LinkedIn, Krishnaram led LinkedIn's efforts in the areas of fairness, transparency, explainability, and privacy in AI/machine learning, and worked with senior leadership across different function areas (AI/ML, Legal, Product, etc.) towards developing and deploying scalable solutions across LinkedIn products.
- Served as LinkedIn's representative in Microsoftā€™s AI and Ethics in Engineering and Research (AETHER) Advisory Board.
- Drafted vision/proposal on fairness/privacy/explainability/trust in AI, and successfully championed for the formation of a new org.
- Tech lead for a virtual team of researchers & engineers working on related areas at LinkedIn.
- Received Distinguished Contribution Award in Microsoft's internal AI/ML conference (MLADS Spring 2018) for our work on detecting and addressing bias in AI systems and designing fairness-aware machine learning/ranking approaches. Video of LinkedIn CEO, Jeff Weiner referring to our work & this award.
- Developed and deployed fairness-aware talent-search system to all users of the LinkedIn Recruiter product worldwide (ACM KDD 2019 paper; LinkedIn Engineering Blog describing the technical architecture).
- Developed and deployed scalable mechanisms for computing privacy-preserving analytics at LinkedIn (ACM CIKM 2018 paper; L David Nigenda, Zohar Karnin, Muhammad Bilal Zafar, Raghu Ramesha, Alan Tan, Michele Donini, Krishnaram Kenthapadi inkedIn Engineering Blog Post).
- Collaborated with Microsoft Research [Papers on fairness in ACM FAT*'19 and NAACL-HLT'19 (Best Thematic Paper Award)].
- Invited to speak at RecSys'19 & several privacy/fairness workshops; Gave invited talks titled "Fairness and Privacy in AI/ML Systems" at QConSF'18 (slides; video), @Scale'19, and Pinterest (slides; video).
- Presented tutorials on Privacy-preserving Data Mining in Industry at KDD 2018 [slides], WSDM 2019 [slides] [proposal], and WWW 2019 [slides] [proposal].
- Presented tutorials on Fairness-aware Machine Learning in Practice at WSDM 2019 [slides] [proposal], WWW 2019 [slides] [proposal], and KDD 2019 [slides].
- Presented tutorial on Explainable AI in Industry at KDD 2019 [slides].
- Filed over 15 patents.
LinkedIn Salary: As the Privacy and Modeling Architect & Lead, Krishnaram shaped the technical roadmap for the LinkedIn Salary product (launched to the public in 2016) and provided technical guidance/leadership since its inception in 2015. The key contributions include:
- Designed the engineering and modeling architecture, taking into account privacy, de-identification, security, accuracy, and scalability requirements, and led a ~15 people team for the privacy/modeling system implementation. This involved balancing the needs of privacy and modeling, and bridging the large gap between the security and engineering teams.
- Designed a privacy-preserving data-processing pipeline to ensure that individual salaries are not discoverable. This involved encrypting/de-identifying different aspects of the data at different stages so that there is no single point of failure, and the design is robust to any potential storage/service breaches.
- Designed and implemented an extensible and robust salary modeling rest.li API, considering the initial product needs (incentive for data collection) as well as subsequent scenarios (basis for LinkedIn Salary product and use as a platform for other LinkedIn verticals). Designed schemas for multiple Voldemort distributed key-value stores to support the Rest.Li interface and the offline aggregation flow. These designs helped bring more stability to the overall software design, which tended to evolve with the UI design.
- Developed and deployed AI/statistical models to measure/improve data quality. The modeling efforts included outlier detection, regression models for estimating compensation, Bayesian hierarchical smoothing, low rank matrix factorization/completion, propensity modeling for bias correction, and addressing numerous conceptual issues such as coverage vs quality trade-off and mapping external datasets to LinkedIn taxonomy.
- Co-authored LinkedIn Engineering Blog Posts titled "The Statistical Modeling System Powering LinkedIn Salary" and "Using Economic Graph Data to Power the LinkedIn Salary Product".
- Published papers on the statistical modeling system (ACM KDD 2018 & ACM CIKM 2017 (Best Case Studies Paper Award)) and the privacy architecture (IEEE PAC 2017).
- Filed 42 patents.
LinkedIn Careers and Talent Solutions AI Initiatives: Krishnaram led the relevance efforts for the Careers and Talent Solutions Relevance team, which powers large scale recommendation/search products at the intersection of members, recruiters, and career opportunities. Key milestones for the 10+ people team:
- Built the recommendation models for the newly launched LinkedIn Referrals product, incorporating models for capturing employee-candidate relationship, job fit, connection fit, etc.
- Built new models for providing flavored job recommendations and associated explanations (jobs in my network, high growth/trending companies, educational edge, etc), thereby personalizing and redefining LinkedIn job seeking experience. Resulted in xx% lift in key metrics such as job application rate.
- Launched latent personalization (seniority and relocation) and activity based job recommendation models (+xx% sitewide job applications / job application rate).
- Launched new models for sponsored job recommendations (doubled CPC revenue) and similar jobs (+xx% job application rate).
- Migrated 10+ production services from legacy recommendation platform to newer offline model training / online recommendation platform.
- Internationalized title standardizer based on machine translated titles, and launched country specific recommendation models in many markets (India, China, ...), with xx% lifts in key metrics.
- Represented the team across organizational boundaries (e.g., product reviews with CEO/executive team, planning with application teams, coordination with other stakeholder teams).
- Cultivated a culture of cross-team & intra-team collaborations, by breaking silos and ensuring collective ownership of relevance efforts.
- Published/presented our work in forums such as KDD'16, KDD'17, SIGIR'17, RecSys'17, CIKM'17, SIGIR'18, KDD'18, and CIKM'18.
- Filed 50+ patent applications.
Document Understanding and Augmentation with Multimedia Content: The rapid proliferation of low-cost tablet devices that are readily connected to the cloud opens up several new opportunities to transform reading experience. With printed books being digitized, as is, we address the following question: How do we enrich the experience of reading from electronic versions of printed books/documents, on cloud-connected tablets? Towards this goal, as a team, we have developed approaches for enriching documents by first understanding the semantics of the content, and using that representation to algorithmically augment a non-fiction book with links to selective web content such as text, images and videos, at any desired granularity. We have incorporated these technologies into a Windows 8 app as well as a web app. These apps are serviced off a Microsoft Azure cloud service that automatically identifies relevant content to augment the book. The app is in the last stage of being released to the Windows App Store. We also transferred the book augmentation technology in the form of the code, necessary data, and documentation to the Office Reader team. Our work enabled the product team to endow the Office Reader with a differentiating capability, and helped enhance the e-reader demos shown to Bill Gates and other top Microsoft executives. (Project page)
Video demonstration of the app
Computational Education: Taking into account the vast amount of existing textbooks designed for traditional printed medium and the potential for enabling new kinds of functionalities through the medium of electronic textbooks, our research focuses on algorithmically diagnosing and enhancing the quality of textbooks. Specifically, we have developed a diagnostic tool for authors and educators to identify weaknesses in a textbook. We have also developed algorithmic and data mining techniques for augmenting different sections of a textbook with links to selective web content (web articles, images & videos) and providing easy access to concepts explained elsewhere in the book that are necessary for understanding the present section. We have built a demonstration of the ideas running on Windows 8 devices as well as on Aakash, the low cost tablet being developed by the Indian government for distribution to millions of students. We showcased our technology and Aakash app (by invitation) to India's Ministry of Human Resource and Development, towards the goal of deploying our app on millions of Aakash tablets. (Project page)
Video demonstration of the app
Algorithmic Identification and Ranking of Twitter Groups: Discovered a new kind of group conversation on Twitter that we call a group chat. These chats are periodic, synchronized group conversations focused on specific topics and they exist at a massive scale. Developed techniques for algorithmically identifying and ranking these groups, and performed large scale evaluation over one year of tweets.
Lincoln: Next Generation Product Search Engine: Built a working end-to-end experimental commerce search engine showcasing state-of-the-art breakthrough technologies in structured search query processing, hierarchical query classification, indexing strategies, catalog creation and decision aids for users to make informed decisions. Received Microsoft Research Tech Transfer Award and Gold Star Award from the product team for transferring and integrating hierarchical query classifier and automatic training data generation technologies as part of Bing Commerce Search product. (Project page)
Privacy in Search Logs: Provided the first solution to a long standing open problem: is there a way to publish internet search logs in a manner that both preserves usefulness and provably protects privacy of the users posing queries. My solution is simple to state and thus practical, yet non-trivial to analyze. Showed how search queries, clicks and their perturbed counts can be released in a privacy-preserving manner and further that the released data can be used for important practical applications such as helping online advertising campaigns and social science research to understand human nature. [WWW 2009 Best Paper Award Finalist]. Subsequently, invented a method for privacy preserving distance estimation that involves projecting each user's representation into a random, lower-dimensional space via a sparse Johnson-Lindenstrauss transform and then adding Gaussian noise to each entry of the lower-dimensional representation. Provided consultation to a product team in Microsoft, highlighting privacy leaks in the data that was planned to be released, thereby saving Microsoft from a huge PR fiasco.
Web Search Ranking: Worked on challenging problems in the areas of relevance and ranking for web search, and engaged with Bing Search product team to transfer the results of the investigations. Examples of investigations included designing a dynamic ranker with a pruned set of most important features, improving the quality of results for specific classes of queries by designing special rankers, and automatic generation of large amounts of training data from search logs. Showed that temporally relevant and user intent aware labeled training data can be generated automatically by aggregating click activity of hundreds of millions of users, by first forming a preference graph using a probabilistic interpretation of the click log and then formulating as a novel graph partitioning problem. This investigation enabled Bing to deploy country/locale specific rankers in over 25 international markets. Received multiple "Ship It" awards for my contributions to web search ranking and relevance improvements in Bing.
Trust and Rating Propagation in Social Networks: Developed random walk based algorithms for trust computation and rating propagation in social networks, implemented over MSN messenger network (consisting of 300M+ nodes), and engaged with product teams across Microsoft to demonstrate the results and to advocate using MSN messenger network and the notion of trust towards providing a differentiating web search experience.

Personal: Photos

Krishnaram Kenthapadi