Round Image

ANDREA BRITTO
MATTOS LIMA

M.Sc. in Computer Science

Senior Research Software Engineer
at Microsoft Research

I hold both B.Sc. and M.Sc. degrees in Computer Science from the Institute of Mathematics and Statistics at the University of Sao Paulo (IME-USP). My research interests include Computer Vision, Machine Learning, and Computer Graphics.

In my professional journey, I worked for two years as an R&D analyst at the Innovation department of IBOPE Media, a Brazilian multinational company. Subsequently, I dedicated eight years to my role as a Research Software Engineer at IBM Research Brazil, where I contributed to projects encompassing computer vision research across various domains.

Presently, I'm working at the Special Projects group at Microsoft Research Lab in Redmond, USA, while being based in Sao Paulo, Brazil. My current research project centers on 3D telemedicine for underserved regions, in collaboration with partners in Scotland and Ghana. Additionally, my team is dedicated to enhancing large language models to improve their effectiveness across different applications.

Resume (PDF)
Background Image

Projects:

Some of my recent and past research initiatives

Project 7

3D Telemedicine

Aims to enhance healthcare accessibility for rural and underserved communities, by leveraging Holoportation™ technology and affordable Azure Kinect sensors. The system captures and streams live 3D content of patients to remote clinicians in real-time, elevating quality and patient satisfaction compared to traditional 2D telemedicine.

Project 6

Paralinguistic Prompting

Focuses on enhancing the capabilities of large language models (LLMs) by addressing their limitations in understanding nonverbal cues and contextual nuances. I worked on machine learning models for recognition of facial expressions - that contribute to meaning and intentions in human interactions.

Project 5

Seismic Interpretation

Created 3D geometry-based post-processing method for Horizon Picking with sematic segmentation networks (client Gazprom Neft) and built experimental framework for evaluating texture descriptors for Seismic Facies Analysis (client Galp Energia).

Project 4

Digital Agriculture

Enhanced crop identification through the evaluation of Convolutional LSTM models' generalization capabilities. Additionally, applied CNNs for the classification of citrus trees using UAV images captured in dense orchards for client 3DGeo.

Project 3

Audio-visual Speech Recognition

Leveraged Generative Adversarial Networks (GANs) to enhance viseme recognition by incorporating multi-view synthetic data.

Project 2

Accessibility

Developed a prototype tailored for individuals with visual impairment and an educational platform designed for those with cognitive disabilities, in collaboration with APAE-SP.

Project 1

Tech for Justice

Conducted a user study to examine potential variations in the career decision-making process across genders and fields (STEM vs. non-STEM). Explored the application of Machine Learning to support underrepresented groups in two distinct tasks: securing micro-credit and accessing qualified job postings.

Publications:

  1. Steven Lo, Anna Rose, Spencer Fowers, Kwame Darko, Andrea Britto Mattos, Thiago Spina, et al. "Ghana 3D Telemedicine International MDT: A Proof-of-concept study", In Journal of Plastic, Reconstructive & Aesthetic Surgery, 2023 (Available at ScienceDirect).
  2. Steven Lo, Spencer Fowers, Kwame Darko, Thiago Spina, Catriona Graham, Andrea Britto Mattos, et al. "Participatory Development of a 3D Telemedicine system during Covid: the future of remote consultations", In Journal of Plastic, Reconstructive & Aesthetic Surgery, 2022 (Available at ScienceDirect).
  3. Amber Hoak, David Tittsworth, Kate Lytvynets, and Andrea Britto Mattos. "Futurizing the Web: Unleashing Creative Possibilities with Browser-Based ML", In Grace Hopper Celebration, 2022 (In-person workshop).
  4. Danilo Calhes, Felipe K. Kobayashi, Andrea Britto Mattos, Maysa M. G. Macedo and Dario A. B Oliveira. “Simplifying Horizon Picking Using Single-Class Semantic Segmentation Networks”, In 34th Sibgrapi – Conference on Graphics, Patterns and Images, 2021 (Available at IEEE Xplore).
  5. Heloisa Candello, Andrea Britto Mattos, Claudio Pinhanez, Rogerio de Paula, and Marcelo Grave. “Microbanking: Bringing Credit to Underrepresented Communities by Enhancing Local Financial Practices With AI”, In AAAI Workshop on Reframing Diversity in AI: Representation, Inclusion and Power, 2020 (Available at CEUR-WS.org).
  6. Maysa M G Macedo, Marisa Vasconcelos, Andrea Britto Mattos, and Rogerio Abreu de Paula. “Connecting Underrepresented Minorities and Qualified Job Positions Using Online Data”, In AAAI Workshop on Reframing Diversity in AI: Representation, Inclusion and Power, 2020 (Available at CEUR-WS.org).
  7. Andrea Britto Mattos, Daniel Civitarese, Daniela Szwarcman, Matheus Oliveira, Semen Zaytsev, Daniil G. Semin, and Dário A. B. Oliveira. “Enabling Robust Horizon Picking From Small Training Sets”, In IEEE Transactions on Geoscience and Remote Sensing, 2020 (Available at IEEE Xplore).
  8. Maysa Malfiza Garcia de Macedo, Andrea Britto Mattos, and Dário Augusto Borges Oliveira. “Generalization of Convolutional LSTM Models for Crop Area Estimation”, In IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2020 (Available at IEEE Xplore).
  9. Felipe Kawashita Kobayashi, Andrea Britto Mattos, Bruno Holtz Gemignani, and Maysa M G Macedo. “Experimental Analysis of Citrus Tree Classification from UAV Images”, In IEEE International Symposium on Multimedia (ISM), 2019 (Available at IEEE Xplore).
  10. Felipe Kawashita Kobayashi, Andrea Britto Mattos, Maysa M G Macedo, and Bruno Holtz Gemignani. “Citrus Tree Classification from UAV Images: Analysis and Experimental Results”, In XV Workshop on Computational Vision (WVC), 2019 (Available at SBC OpenLib).
  11. Dario Augusto Borges Oliveira, Andrea Britto Mattos, and Edmilson da Silva Morais. "Improving Viseme Recognition with GAN-based Muti-view Mapping", In Proceedings of the IEEE Conference on Automatic Face and Gesture Recognition (FG), 2019 (Available at IEEE Xplore).
  12. Andrea Britto Mattos, Dario Augusto Borges Oliveira, and Edmilson da Silva Morais. "Improving Viseme Recognition with Synthetic Data using CNNs and GAN-based Muti-view Mapping", In 13th Women in Machine Learning Workshop (WiML), 2018.
  13. Maciel Zortea, Maysa M. G. Macedo, Andrea Britto Mattos, Bernardo C. Ruga, Bruno H. Gemignani. "Automatic citrus tree detection from UAV images based on convolutional neural networks", In 31th Sibgrap/WIA - Conference on Graphics, Patterns and Images, 2018 (Available at ResearchGate).
  14. Andrea Britto Mattos, Dario Augusto Borges Oliveira, and Edmilson da Silva Morais. "Towards View-independent Viseme Recognition based on CNNs and Synthetic Data", In Proceedings of the IEEE International Conference on Image Processing (ICIP), 2018 (Available at IEEE Xplore)
  15. Maysa Macedo, Andrea Britto Mattos, and Marisa Vaconcelos. "Identificando Influências na Escolha de uma Graduação em Exatas: um Estudo Qualitativo e Comparativo de Gênero", In 12th Women in Information Technology (WIT-CSBC), 2018 (Available at ResearchGate)
  16. Andrea Britto Mattos, Dario Augusto Borges Oliveira, and Edmilson da Silva Morais. "Improving CNN-based Viseme Recognition Using Synthetic Data", In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), 2018 (Available at IEEE Xplore)
  17. Dario Augusto Borges Oliveira, Andrea Britto Mattos, and Edmilson da Silva Morais. "Improving Viseme Recognition using GAN-based Frontal View Mapping", In Analysis and Modeling of Faces and Gestures (AMFG@CVPR), 2018 (Available at OpenAccess)
  18. Andrea Britto Mattos and Dario Augusto Borges Oliveira. "Multi-view Mouth Renderization for Assisting Lip-reading", In Proceedings of the 15th Web for All Conference, 2018. (Available at ACM DL)
  19. Emilio Vital Brazil, Rodrigo da Silva Ferreira, Andrea Britto Mattos, Reinaldo M. Da Gama e Silva, Renato Cerqueira, Marco Ferraz, and Sergio Cersosimo. "Assessment of Texture Descriptors for Seismic Image Retrieval and Salt Dome Detection", In Proceedings of the 8th International Geological and Geophysical Conference and Exhibition (EAGE), 2018. (Available at EarthDoc)
  20. Andrea Britto Mattos, Rodrigo da Silva Ferreira, Reinaldo M. Da Gama e Silva, Mateus Riva, and Emilio Vital Brazil. "Assessing Texture Descriptors for Seismic Image Retrieval", In 30th Sibgrapi - Conference on Graphics, Patterns and Images, 2017. (Available at IEEE Xplore)
  21. Rodrigo Laiola Guimaraes, Andrea Britto Mattos, and Carlos Cardonha. "A Comparative Study of Technology-Mediated Learning Strategies for Teaching People with Intellectual Disability", In Proceedings of the 16th Brazilian Symposium on Human Factors in Computer Systems, 2017. (Available at ACM DL)
  22. Rodrigo Laiola Guimaraes, Andrea Britto Mattos, and Carlos Cardonha. "Estudo Qualitativo do Uso de Apoios Tecnológicos no Ensino de Pessoas com Deficiência Intelectual", In Proceedings of the 16th Brazilian Symposium on Human Factors in Computer Systems, 2017.
  23. Rodrigo da Silva Ferreira, Andrea Britto Mattos, Emilio Vital Brazil, Renato Cerqueira, Marco Ferraz, and Sergio Cersosimo. "Multi-scale Evaluation of Texture Features for Salt Dome Detection", In IEEE International Symposium on Multimedia (ISM), 2016. (Available at IEEE Xplore)
  24. Rodrigo Laiola Guimaraes, Andrea Britto Mattos, and Carlos Cardonha. "Investigating Instructional Pacing Supports for Teaching Students with Intellectual Disability", In Proceedings of the CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2016. (Available at ACM DL)
  25. Carlos Cardonha, Andrea Britto Mattos, and Rodrigo Laiola Guimaraes. "A Platform to Support Personalized Training for People with Disabilities", In Proceedings of the 13th Web for All Conference, 2016. (Available at ACM DL)
  26. Vagner Figueredo de Santana, Rodrigo Laiola Guimaraes, and Andrea Britto Mattos. "Identifying Challenges and Opportunities in Computer-based Vocational Training for Low-Income Communities of People with Intellectual Disabilities", In Proceedings of the 13th Web for All Conference, 2016. (Available at ACM DL)
  27. Carlos Henrique Cardonha, Rodrigo Laiola Guimaraes, Andrea Britto Mattos, Julio Nogima, Priscilla Avegliano, Diego Gallo, Ricardo Herrmann, and Sergio Borger. "Toward a platform to support vocational training of people with disabilities", In IBM Journal of Research and Development, 2015. (Available at IEEE Xplore)
  28. Rodrigo Laiola Guimaraes and Andrea Britto Mattos. "Exploring the Use of Massive Open Online Courses for Teaching Students with Intellectual Disability", In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS), 2015. (Available at ACM DL)
  29. Rodrigo Laiola Guimaraes, Andrea Britto Mattos, Victor Martinez, and Flavio Gonzalez. "Apoio Tecnologico para Treinamento Vocacional", In Revista Deficiencia Intelectual, APAE de Sao Paulo, 2014. (Available at ResearchGate)
  30. Andrea Britto Mattos and Rogerio Schmidt Feris. "Fusing well-crafted feature descriptors for efficient fine-grained classification", In 21th IEEE International Conference on Image Processing (ICIP), 2014. (Available at IEEE Xplore)
  31. Andrea Britto Mattos, Ricardo Guimaraes Herrmann, Kelly Kiyumi Shigeno, and Rogerio Schmidt Feris. "A Mission-Oriented Citizen Science Platform for Efficient Flower Classification Based on Combination of Feature Descriptors", In International Workshop on Environmental Multimedia Retrieval, 2014. (Available at ResearchGate)
  32. Andrea Britto Mattos, Ricardo Guimaraes Herrmann, Kelly Kiyumi Shigeno, and Rogerio Schmidt Feris. "Flower Classification for a Citizen Science Mobile App", In Proceedings of International Conference on Multimedia Retrieval, 2014. (Available at ACM DL)
  33. Andrea Britto Mattos, Ricardo Herrmann, Carlos Cardonha, Diego Gallo, Priscilla Avegliano, and Sergio Borger. "Marker-assisted recognition of dynamic content in public spaces", In Proceedings of the 11th Web for All Conference, 2014. (Available at ACM DL)
  34. Andrea Britto Mattos, Carlos Cardonha, Diego Gallo, Priscilla Avegliano, Ricardo Herrmann, and Sergio Borger. "Marker-based image recognition of dynamic content for the visually impaired", In Proceedings of the 11th Web for All Conference, 2014. (Available at ACM DL)
  35. Priscilla Avegliano, Carlos Cardonha, Andrea Britto Mattos, and Sergio Borger. "Shelf replenishment optimization towards the confluence of mobile and analytics", In 14th Annual Conference on Business Analytics (INFORMS), 2014.
  36. Andrea Britto Mattos, Marcelo Van Kampen, Camila Carrico, Andre Ricardo Dias, and Alexandre Crivellaro. "E-commerce market analysis from a graph-based product classifier", In Computational Processing of the Portuguese Language (PROPOR), 2012. (Available at SpringerLink)
  37. Andrea Britto Mattos, Jesus P Mena-Chalco, and Roberto Marcondes Cesar Jr. "3D facial animation based on structural registration", In 24th Sibgrapi (Conference on Graphics, Patterns and Images) - Workshop of Theses and Dissertations, 2011. (Available at ResearchGate)
  38. Andrea Britto Mattos, Igor dos Santos Montagner, Alexandre Crivellaro, Bruno Klava, and Marcel Brun. "Computer vision research at IBOPE Media: automation tools to reduce human intervention", In 24th Sibgrapi (Conference on Graphics, Patterns and Images) - Workshop on Interactive Graphics and Visualization for Industry-Driven Applied and Exploratory Research, 2011. (Available at ResearchGate)
  39. Andrea Britto Mattos, Jesus P Mena-Chalco, Roberto Marcondes Cesar Jr, and Luiz Velho. "3D linear facial animation based on real data", In 23rd Sibgrapi (Conference on Graphics, Patterns and Images), 2010. (Available at IEEE Xplore)
  40. Andrea Britto Mattos and Roberto Marcondes Cesar Jr. "3D facial animation based on structural registration", In Workshops of Sibgrapi Posters, 2009. (Available at ResearchGate)

Patents:

  • Adjusting manufacturing parameters using artificial intelligence.
  • System for optimizing sensor settings in a multi-camera environment based on foundation models and historical data.
  • Enhanced user interfaces for paralinguistics.
  • Augmenting artificial intelligence prompt design with emotional context.
  • Real-time system for streaming with live volumetric capture and associated metadata.
  • Flap prediction system based on volumetric data and foundation models.
  • Training and implementing a four-dimensional data object recommendation model.
  • US20220198221A1 - Artificial intelligence generated synthetic image data for use with machine language models.
  • US20210218695A1 - Adapting communications according to audience profile from social media.
  • US20210064651A1 - Searching for analogue subsurface structures based on topological knowledge representation (TKR).
  • US20210048303A1 - Real-time route determination based on physiological responses.
  • US20210034717A1 - Media transactions consent management.
  • US20200353622A1 - Online utility-driven spatially-referenced data collector for classification.
  • US10924661B2 - Generating image capture configurations and compositions.
  • US11138302B2 - Access control using multi-authentication factors.
  • US20200241524A1 - Intelligent area and dispersal management using autonomous vehicles.
  • US20200221754A1 - Image-based encoding.
  • US 20200120267A1 - Advising image acquisition based on existing training sets.
  • US 10769466B2 - Precision aware drone-based object mapping based on spatial pattern recognition.
  • US 20162346A1 - Recommending activity sensor usage by image processing.
  • US 20161728A1 - Processing image using narrowed search space based on textual context to detect items in the image.
  • US 20161724A1 - Learning systems and automatic transitioning between learning systems.
  • US 20160767A1 - Adapting physical activities and exercises based on facial analysis by image processing.
  • US 20160023A1 - System and method for identification of personal thermal comfort.
  • US 20151195A1 - System and method for automatic identification of review material.
  • US 20151286A1 - Detection of anomalous behavior in digital education settings based on portable device movement.
  • US 20151373A1 - Educational media planning and delivery for in-class lessons with limited duration.
  • US 20151202A1 - Restocking shelves based on image data.
  • US 20150294333A1 - Mobile device based inventory management and sales trends analysis in a retail environment.
  • BR 10 2012 009983 7 A2 - Método de estimação de audiência e monitoramento de navegação na internet.

Master thesis:

Animação de faces tridimensionais baseada em registro estrutural

"3D linear facial animation based on structural registration" (Defended on May, 2011).

My master thesis focuses on computer facial animation, three-dimensional registration, and structural graph matching. I was advised by Prof Roberto M. Cesar Jr. and my work was supported by FAPESP. I was co-advised by Prof Luiz Velho and a collaborator of the VISGRAF Lab – IMPA/RJ. The materials are availabe below for download (in Portuguese).

Full text (PDF)

Your Image

University Talks:

  • UFABC: "Machine Learning para Visão Computacional", at IV Semana das Engenharias, 2018 - Santo Andre/SP, Brazil.
  • UNICAMP: Andrea Britto Mattos Lima, Heloisa Candello and Alan Braz. "Pesquisa aplicada na indústria de tecnologia", at Computer Science Seminar Series, 2016 - Sao Paulo/SP, Brazil.
  • UFSCAR: "Cidades Inteligentes: promovendo acessibilidade e educação", at 2nd Forum of the City Institute, 2014 - Sao Carlos/SP, Brazil.
  • UNIP: "Sistemas de engajamento para acessibilidade", at Computer Science Seminar Series, 2014 - Sao Paulo/SP, Brazil.
  • UFABC: "Plataformas para acessibilidade e educação", at Computer Science Seminar Series, 2014 - Santo Andre/SP, Brazil.
  • IME/USP: "Pesquisa aplicada em Visão Computacional para o IBOPE Media", at Seminars of the Computer Vision Research Group, 2011 - Sao Paulo/SP, Brazil.

Other Talks:

  • Women in Data Science (WIDS) São Paulo: "Painel sobre Carreira em Ciência de Dados", Invite panelist, 2023 - Sao Paulo/SP, Brazil.
  • Women in Data Science (WIDS) UNESP: "Machine Learning na prática: da agricultura a mudanças climáticas", 2021 - Online event.
  • AI Girls: “Usando Machine learning e Visão computacional para resolver problemas reais: minha experiência na IBM Research”, 2021 - Online event (Available at Youtube).
  • Bluetalks | inovabra: "Crédito para comunidades minoritárias através da IA", 2021 - Online event.
  • IBM TLC-Tech Day: "AI, Watson, & Agro", 2019 - Hortolandia/SP, Brazil.
  • 11th Women in Information Technology (WIT-CSBC): "Mulheres na IBM: Oportunidades, carreiras técnicas e pesquisa", Invited speaker, 2017 - Sao Paulo/SP, Brazil.
  • III Latin-American School on Software Engineering (ELA-ES): "Women into Computing", Invited panelist, 2016 - Natal/RN, Brazil.
  • IBM Thomas J. Watson Research Center: “Accessibility projects in BRL and potential directions in the multimedia field”, 2015 - Yorktown Heights/NY, USA.
  • Feira Inovatec: "IBOPE - políticas de parcerias e áreas de interesse em inovação aberta", 2011 - Sao Paulo/SP, Brazil.

Conferences:

List of conferences where I participated in as a presenter or attendee

  • Conference & Exhibition on Computer Graphics & Interactive Techniques (SIGGRAPH), 2023 - Los Angeles, California.
  • Conference & Exhibition on Computer Graphics & Interactive Techniques (SIGGRAPH), 2022 - Virtual.
  • Conference & Exhibition on Computer Graphics & Interactive Techniques in Asia (SIGGRAPH ASIA), 2021 - Virtual.
  • Conference on Neural Information Processing Systems (NeurIPS), 2018 - Montreal, Canada.
  • International Conference on Image Processing (ICIP), 2018 - Athens, Greece.
  • International Conference on Multimedia and Expo (ICME), 2018 - San Diego, California.
  • The Web for All Conference (W4A), 2018 - Lyon, France.
  • Global Symposium on AI & Inclusion, 2017 - Rio de Janeiro/RJ, Brazil.
  • Conference on Graphics, Patterns and Images (Sibgrapi), 2017 - Niterói/RJ, Brazil.
  • The Web for All Conference (W4A), 2016 - Montreal, Canada.
  • IBM First Deep Learning Workshop, 2016 - Yorktown Heights/NY - USA.
  • IBM WW Accessibility, 2015 - Boston/MA, USA.
  • The International Conference on Image Processing (ICIP), 2014 - Paris, France.
  • The Web for All Conference (W4A), 2014 - Seoul, South Korea.
  • International Conference on Multimedia Retrieval (ICMR), 2014 - Glasgow, Scotland.
  • Computational Processing of the Portuguese Language (PROPOR), 2012 - Coimbra, Portugal.
  • Open Innovation Seminar, 2011 - Sao Paulo/SP, Brazil.
  • Conference on Graphics, Patterns and Images (Sibgrapi), 2011 - Maceió/AL, Brazil.
  • Conference on Graphics, Patterns and Images (Sibgrapi), 2010 - Gramado/RS, Brazil.
  • Conference on Graphics, Patterns and Images (Sibgrapi), 2009 - Rio de Janeiro/RJ, Brazil.

List of conferences where I served on the organizing committee

  • Symposium on Knowledge Discovery, Mining and Learning (KDMiLe 2018): Local chair, Sao Paulo/SP, Brazil.
  • Sao Paulo Advanced School of Computing (SP-ASC 2010): Member of the local committee, Sao Paulo/SP, Brazil.
  • 15th Iberoamerican Congress on Pattern Recognition (CIARP 2010): Member of the local committee, Sao Paulo/SP, Brazil.
Background Image

Press:

  • MarkTechPost: Meet Project Rumi: Multimodal Paralinguistic Prompting for Large Language Models. [link]
  • FutureScot: Scotland's 'innovation fellowship scheme' points to better health and social care future. [link]
  • BBC Scotland: 3D Telemedicine Project aired at Reporting Scotland in July/2022.
  • Forbes: A new technology to help visually impaired shoppers is being tested by IBM in Brazil. [link]
  • Chain Store Age: IBM creates digital marketing tool for visually impaired. [link]
  • Revista Pesquisa FAPESP: Olhar eletrônico. [link]

Honors and Achievements:

  • Best paper nomination on IEEE International Symposium on Multimedia (ISM), 2019.
  • Best paper award on XV Workshop on Computational Vision (WVC), 2019.
  • IBM Fifth Plateau Invention Achievement Award, 2019.
  • IBM Fourth Plateau Invention Achievement Award, 2019.
  • IBM Third Plateau Invention Achievement Award, 2019.
  • IBM Research Accomplishment Award: "Research Contributions to Accessibility and Education", 2018.
  • IBM Research Accomplishment Award: "Cognitive Computing for Seismic Interpretation", 2017.
  • Honorable mention at IHC Conference (track HCI in practice), 2017.
  • IBM Second Plateau Invention Achievement Award, 2017.
  • Best paper nomination at Web for All Conference (W4A), 2016.
  • IBM First Plateau Invention Achievement Award, 2016.
  • Best paper award on The Paciello Group Web Accessibility Challenge (W4A), 2014.
  • IBM Research Accomplishment Award: "Mobile based crowdsourcing platform for consumer and citizen engagement and insight", 2014.
  • FAPESP scholarship grant (May/2009 - March/2011).

Volunteering experience:

  • Ignite Worldwide (2023): How to Get a Job in Technology Work-Based Learning Experience with Microsoft. [link]
  • Ignite Worldwide (2022): Cybersecurity Work-Based Learning Experience with Microsoft. [link]
  • Adote um Gatinho (2019): The NGO specializes in rescuing street cats and facilitating their adoption. I contributed to the team responsible for transporting cats from the shelter to adopters' homes, ensuring safety measures, and managing contractual matters.
  • Paulo Montenegro Institute (2012): Teaching basic math and statistic lessons for adults of low-income communities.

Last update: March/2024

Background Image