Vasant G. Honavar is an Indian-American computer scientist, and artificial intelligence, machine learning, big data, data science, causal inference, knowledge representation, bioinformatics and health informatics researcher and professor. == Early life and education == Vasant Honavar was born at Pune, India to Bhavani G. and Gajanan N. Honavar. He received his early education at the Vidya Vardhaka Sangha High School and M.E.S. College in Bangalore, India. He received a B.E. in Electronics & Communications Engineering from the B.M.S. College of Engineering in Bangalore, India in 1982, when it was affiliated with Bangalore University, an M.S. in electrical and computer engineering in 1984 from Drexel University, and an M.S. in computer science in 1989, and a Ph.D. in 1990, respectively, from the University of Wisconsin–Madison, where he studied Artificial Intelligence and worked with Leonard Uhr. == Career == Honavar is on the faculty of Informatics and Intelligent Systems Department in the Penn State College of Information Sciences and Technology at Pennsylvania State University where he currently holds the Dorothy Foehr Huck and J. Lloyd Huck Chair in Biomedical Data Sciences and Artificial Intelligence and previously held the Edward Frymoyer Endowed Chair in Information Sciences and Technology. He serves on the faculties of the graduate programs in Computer Science, Informatics, Bioinformatics and Genomics, Neuroscience, Operations Research, Public Health Sciences, and of undergraduate programs in Data Science and Artificial Intelligence methods and applications. Honavar serves as the director of the Artificial Intelligence Research Laboratory, Director of Strategic Initiatives for the Institute for Computational and Data Sciences and the director of the Center for Artificial Intelligence Foundations and Scientific Applications at Pennsylvania State University. Honavar served on the Leadership Team of the Northeast Big Data Innovation Hub. Honavar served on the Computing Research Association's Computing Community Consortium Council during 2014-2017, where he chaired the task force on Convergence of Data and Computing, and was a member of the task force on Artificial Intelligence. Honavar was the first Sudha Murty Distinguished Visiting Chair of Neurocomputing and Data Science by the Indian Institute of Science, Bangalore, India. Honavar was named a Distinguished Member of the Association for Computing Machinery for "outstanding scientific contributions to computing"; and elected a Fellow of the American Association for the Advancement of Science for his "distinguished research contributions and leadership in data science". As a Program Director in the Information Integration and Informatics program in the Information and Intelligent Systems Division of the Computer and Information Science and Engineering Directorate of the US National Science Foundation during 2010-13, Honavar led the Big Data Program. Honavar was a professor of computer science at Iowa State University where he led the Artificial Intelligence Research Laboratory which he founded in 1990 and was instrumental in establishing an interdepartmental graduate program in Bioinformatics and Computational Biology (and served as its Chair during 2003–2005). Honavar has held visiting professorships at Carnegie Mellon University, the University of Wisconsin–Madison, and at the Indian Institute of Science. == Research == Honavar's research has contributed to advances in artificial intelligence, machine learning, causal inference, knowledge representation, neural networks, semantic web, big data analytics, and bioinformatics and computational biology. He was a program chair of the Association for the Advancement of Artificial Intelligence(AAAI)'s 36th Conference on Artificial Intelligence. He has published over 300 research articles, including many highly cited ones, as well as several books on these topics. His recent work has focused on federated machine learning algorithms for constructing predictive models from distributed data and linked open data, learning predictive models from high dimensional longitudinal data, reasoning with federated knowledge bases, detecting algorithmic bias, big data analytics, analysis and prediction of protein-protein, protein-RNA, and protein-DNA interfaces and interactions, social network analytics, health informatics, secrecy-preserving query answering, representing and reasoning about preferences, and causal inference from complex, e.g., relational, data, large language models, diffusion models, and meta analysis. Honavar has been active in fostering national and international scientific collaborations in Artificial Intelligence, Data Sciences, and their applications in addressing national, international, and societal priorities in accelerating science, improving health, transforming agriculture through partnerships that bring together academia, non-profits, and industry. He is also active in making the science policy case for major national research initiatives such as AI for accelerating science and AI for combating the epidemic of diseases of despair. == Honors == National Science Foundation Director's Award for Superior Accomplishment, 2013 National Science Foundation Director's Award for Collaborative Integration, 2012 Margaret Ellen White Graduate Faculty Award, Iowa State University, 2011 Outstanding Career Achievement in Research Award, College of Liberal Arts and Sciences, Iowa State University, 2008 Regents Award for Faculty Excellence, Iowa Board of Regents, 2007 Edward Frymoyer Endowed Chair in Information Sciences and Technology, Penn State College of Information Sciences and Technology, Pennsylvania State University, 2013 Senior Faculty Research Excellence Award, Penn State College of Information Sciences and Technology, Pennsylvania State University, 2016 125 People of Impact, Department of Electrical and Computer Engineering, University of Wisconsin-Madison, 2016 Sudha Murty Distinguished (Visiting) Chair of Neurocomputing and Data Science, Indian Institute of Science, 2016-2021 ACM Distinguished Member, 2018 AAAS Fellow American Association for the Advancement of Science, 2018 EAI Fellow European Alliance for Innovation, 2019 Dorothy Foehr Huck and J. Lloyd Huck Chair in Biomedical Data Sciences and Artificial Intelligence, Pennsylvania State University, 2021
GeneTalk
GeneTalk is a web-based platform, tool, and database for filtering, reduction and prioritization of human sequence variants from next-generation sequencing (NGS) data. GeneTalk allows editing annotation about sequence variants and build up a crowd sourced database with clinically relevant information for diagnostics of genetic disorders. GeneTalk allows searching for information about specific sequence variants and connects to experts on variants that are potentially disease-relevant. == Application to diagnostics == Users can upload NGS data in Variant Call Format (VCF) onto the GeneTalk server into their accounts. All entries of the file are preprocessed and shown in the integrated VCF viewer. Filtering tools are set by the user to reduce the number of clinically non-relevant variants. After filtering and prioritization users can interpret relevant variants by retrieving information (annotations) about variants from the GeneTalk database. The communication platform allow users to contact experts about specific variants, genes, or genetic disorders, to exchange knowledge and expertise. === Analysis procedure === Steps required to analyze VCF files Upload VCF file Edit pedigree and phenotype information for segregation filtering Filter VCF file by editing the filtering options View results and annotations Add annotations === Filtering tools === The following filtering options may be used to reduce the non-relevant sequence variants in VCF files. Functional – filter out variants that have effects on protein level Linkage – filter out variants that are on specified chromosomes Gene panel – filter variants by genes or gene panels, subscribe to publicly available gene panels or create own ones Frequency – show only variants with a genotype frequency lower than specified Inheritance – filter out variants by presumed mode of inheritance Annotation – show only variants with a score for medical relevance and scientific evidence == Communication platform and expert network == Users can share VCF files with colleagues and coworkers. The integrated mailing systems allows users to contact experts easily. Users can create annotations and comments and rate annotations regarding medical relevance and scientific evidence, that is helpful for the community of users for diagnosis of genetic disorders. Registered users provide information about their field of knowledge in their profile and can be contacted by other users. == Potential applications == Developing diagnostics Genetic analysis Capturing data generated by community Communication and exchange of knowledge and expertise
Resistance Database Initiative
HIV Resistance Response Database Initiative (RDI) was formed in 2002 to use artificial intelligence (AI) to predict how patients will respond to HIV drugs using data from more 250,000 patients from around 50 countries around the world. The RDI used its models to power its HIV Treatment Response Prediction System (HIV-TRePS). Launched in 2010, this free online tool enabled healthcare professionals to upload their patient’s data and obtain highly accurate predictions of how they would respond to different combinations of the 30 or more drugs available. The tool enabled physicians to individualize their patients’ treatment, using these predictions based on more than a million patient-years of treatment experience. HIV-TRePS was possibly the first ever AI-based system for medical decision-making to be developed, successfully tested, and used in clinical practice. It has since been used by thousands of healthcare professionals to optimise the treatment of tens of thousands of patients. Since the RDI’s inception the treatment of HIV infection has progressed enormously, with more effective and better tolerated drugs available in ever more convenient combination formulations. In most countries HIV is now considered a chronic, manageable condition. Moreover, the success of the drugs in reducing the amount of virus is substantially reducing the onward transmission of the virus and cases of new infections are falling in many settings. This improvement in HIV treatment means the need for sophisticated AI to support HIV treatment decisions has significantly reduced. In response, the RDI ceased development of further models and, in March 2024, withdrew its HIV-TRePS system. == Background == Human immunodeficiency virus (HIV) is the virus that causes acquired immunodeficiency syndrome (AIDS), a condition in which the immune system begins to fail, leading to life-threatening opportunistic infections. There are approximately 30 HIV antiretroviral drugs that have been approved for the treatment of HIV infection, from six different classes, based on the point in the HIV life-cycle at which they act. They are used in combination; typically 3 or more drugs from 2 or more different classes, a form of therapy known as highly active antiretroviral therapy (HAART). The aim of therapy is to suppress the virus to very low, ideally undetectable, levels in the blood. This prevents the virus from depleting the immune cells that it preferentially attacks CD4 cells and prevents or delays illness and death. Despite the expanding availability of these drugs and the impact of their use, treatments continue to fail, often involving to the development of resistance. During drug therapy, low-level virus replication may still occur, particularly when a patient misses a dose. HIV makes errors in copying its genetic material and, if a mutation makes the virus resistant to one or more of the drugs in the patient's treatment, it may begin to replicate more successfully in the presence of that drug and undermine the effect of the treatment. If this happens, the treatment needs to be changed to re-establish control over the virus. == RDI's Approach == The RDI’s approach was to use artificial intelligence (including neural network and random forest models), trained with data from hundreds of thousands of patients, treated with different drugs in a variety of clinical settings all over the world, to predict how an individual patient will respond to any new combination of HIV drugs. The models were tested with independent data sets and consistently achieved accuracy of approximately 80%.
Probabilistic database
Most real databases contain data whose correctness is uncertain. In order to work with such data, there is a need to quantify the integrity of the data. This is achieved by using probabilistic databases. A probabilistic database is an uncertain database in which the possible worlds have associated probabilities. Probabilistic database management systems are currently an active area of research. "While there are currently no commercial probabilistic database systems, several research prototypes exist..." Probabilistic databases distinguish between the logical data model and the physical representation of the data much like relational databases do in the ANSI-SPARC Architecture. In probabilistic databases this is even more crucial since such databases have to represent very large numbers of possible worlds, often exponential in the size of one world (a classical database), succinctly. == Terminology == In a probabilistic database, each tuple is associated with a probability between 0 and 1, with 0 representing that the data is certainly incorrect, and 1 representing that it is certainly correct. === Possible worlds === A probabilistic database could exist in multiple states. For example, if there is uncertainty about the existence of a tuple in the database, then the database could be in two different states with respect to that tuple—the first state contains the tuple, while the second one does not. Similarly, if an attribute can take one of the values x, y or z, then the database can be in three different states with respect to that attribute. Each of these states is called a possible world. Consider the following database: (Here {b3, b3′, b3′′} denotes that the attribute can take any of the values b3, b3′ or b3′′) Assuming that there is uncertainty about the first tuple, certainty about the second tuple, and uncertainty about the value of attribute B in the third tuple. Then the actual state of the database may or may not contain the first tuple (depending on whether it is correct or not). Similarly, the value of the attribute B may be b3, b3′ or b3′′. Consequently, the possible worlds corresponding to the database are as follows: === Types of Uncertainties === There are essentially two kinds of uncertainties that could exist in a probabilistic database, as described in the table below: By assigning values to random variables associated with the data items, different possible worlds can be represented. == History == The first published use of the term "probabilistic database" was probably in the 1987 VLDB conference paper "The theory of probabilistic databases", by Cavallo and Pittarelli. The title (of the 11 page paper) was intended as a bit of a joke, since David Maier's 600 page monograph, The Theory of Relational Databases, would have been familiar at that time to many of the conference participants and readers of the conference proceedings.
Xaitment
xaitment is a German-based company that develops and sells artificial intelligence (AI) software to video game developers and simulation developers. The company was founded in 2004 by Dr. Andreas Gerber, and is a spin-off of the German Research Centre for Artificial Intelligence, or DFKI. xaitment has its main office in Quierschied, Germany, and field offices in San Francisco and China. == Products == xaitment currently sells two AI software modules: xaitMap and xaitControl. xaitMap provides runtime libraries and graphical tools for navigation mesh generation (also called NavMesh generation), pathfinding, dynamic collision avoidance, and individual and crowd movement. xaitControl is a finite-state machine for game logic and character behavior modeling that also includes a real-time debugger. On January 11, 2012, xaitment announced that it making its source code for these modules available to "all current and future US and European licensees". On February 22, 2012 xaitment released two new plug-ins, xaitMap and xaitControl for the Unity Game Engine. The full versions are available for PC (Windows and Linux), PlayStation 3, Xbox 360 and Wii. The pathfinding plug-in is available with a Windows dev environment, but can deployed on iOS, Mac, Android and the Unity Web Player. == Partners == xaitment's AI software is currently integrated into the Unity game engine, Havok's Vision Engine, Bohemia Interactive's VBS2 Simulation Engine, GameBase's Gamebryo game engine. == Customers == xaitment sells its AI software products to video game developers and military and civil simulation developers. Current customers include Tencent, gamania, TML Studios, Emobi Games, IP Keys and others. A full list of customers can be found on xaitment's website.
Voice activity detection
Voice activity detection (VAD), also known as speech activity detection or speech detection, is the detection of the presence or absence of human speech, used in speech processing. The main uses of VAD are in speaker diarization, speech coding and speech recognition. It can facilitate speech processing, and can also be used to deactivate some processes during non-speech section of an audio session: it can avoid unnecessary coding/transmission of silence packets in Voice over Internet Protocol (VoIP) applications, saving on computation and on network bandwidth. VAD is an important enabling technology for a variety of speech-based applications. Therefore, various VAD algorithms have been developed that provide varying features and compromises between latency, sensitivity, accuracy and computational cost. Some VAD algorithms also provide further analysis, for example whether the speech is voiced, unvoiced or sustained. Voice activity detection is usually independent of language. It was first investigated for use on time-assignment speech interpolation (TASI) systems. == Algorithm overview == The typical design of a VAD algorithm is as follows: There may first be a noise reduction stage, e.g. via spectral subtraction. Then some features or quantities are calculated from a section of the input signal. A classification rule is applied to classify the section as speech or non-speech – often this classification rule finds when a value exceeds a certain threshold. There may be some feedback in this sequence, in which the VAD decision is used to improve the noise estimate in the noise reduction stage, or to adaptively vary the threshold(s). These feedback operations improve the VAD performance in non-stationary noise (i.e. when the noise varies a lot). A representative set of recently published VAD methods formulates the decision rule on a frame by frame basis using instantaneous measures of the divergence distance between speech and noise. The different measures which are used in VAD methods include spectral slope, correlation coefficients, log likelihood ratio, cepstral, weighted cepstral, and modified distance measures. Independently from the choice of VAD algorithm, a compromise must be made between having voice detected as noise, or noise detected as voice (between false positive and false negative). A VAD operating in a mobile phone must be able to detect speech in the presence of a range of very diverse types of acoustic background noise. In these difficult detection conditions it is often preferable that a VAD should fail-safe, indicating speech detected when the decision is in doubt, to lower the chance of losing speech segments. The biggest difficulty in the detection of speech in this environment is the very low signal-to-noise ratios (SNRs) that are encountered. It may be impossible to distinguish between speech and noise using simple level detection techniques when parts of the speech utterance are buried below the noise. == Applications == VAD is an integral part of different speech communication systems such as audio conferencing, echo cancellation, speech recognition, speech encoding, speaker recognition and hands-free telephony. In the field of multimedia applications, VAD allows simultaneous voice and data applications. Similarly, in Universal Mobile Telecommunications Systems (UMTS), it controls and reduces the average bit rate and enhances overall coding quality of speech. In cellular radio systems (for instance GSM and CDMA systems) based on Discontinuous Transmission (DTX) mode, VAD is essential for enhancing system capacity by reducing co-channel interference and power consumption in portable digital devices. In speech processing applications, voice activity detection plays an important role since non-speech frames are often discarded. For a wide range of applications such as digital mobile radio, Digital Simultaneous Voice and Data (DSVD) or speech storage, it is desirable to provide a discontinuous transmission of speech-coding parameters. Advantages can include lower average power consumption in mobile handsets, higher average bit rate for simultaneous services like data transmission, or a higher capacity on storage chips. However, the improvement depends mainly on the percentage of pauses during speech and the reliability of the VAD used to detect these intervals. On the one hand, it is advantageous to have a low percentage of speech activity. On the other hand, clipping, that is the loss of milliseconds of active speech, should be minimized to preserve quality. This is the crucial problem for a VAD algorithm under heavy noise conditions. === Use in telemarketing === One controversial application of VAD is in conjunction with predictive dialers used by telemarketing firms. In order to maximize agent productivity, telemarketing firms set up predictive dialers to call more numbers than they have agents available, knowing most calls will end up in either "Ring – No Answer" or answering machines. When a person answers, they typically speak briefly ("Hello", "Good evening", etc.) and then there is a brief period of silence. Answering machine messages are usually 3–15 seconds of continuous speech. By setting VAD parameters correctly, dialers can determine whether a person or a machine answered the call and, if it's a person, transfer the call to an available agent. If it detects an answering machine message, the dialer hangs up. Often, even when the system correctly detects a person answering the call, no agent may be available, resulting in a "silent call". Call screening with a multi-second message like "please say who you are, and I may pick up the phone" will frustrate such automated calls. == Performance evaluation == To evaluate a VAD, its output using test recordings is compared with those of an "ideal" VAD – created by hand-annotating the presence or absence of voice in the recordings. The performance of a VAD is commonly evaluated on the basis of the following four parameters: FEC (Front End Clipping): clipping introduced in passing from noise to speech activity; MSC (Mid Speech Clipping): clipping due to speech misclassified as noise; OVER: noise interpreted as speech due to the VAD flag remaining active in passing from speech activity to noise; NDS (Noise Detected as Speech): noise interpreted as speech within a silence period. Although the method described above provides useful objective information concerning the performance of a VAD, it is only an approximate measure of the subjective effect. For example, the effects of speech signal clipping can at times be hidden by the presence of background noise, depending on the model chosen for the comfort noise synthesis, so some of the clipping measured with objective tests is in reality not audible. It is therefore important to carry out subjective tests on VADs, the main aim of which is to ensure that the clipping perceived is acceptable. In VoIP applications, front-end clipping can be reduced by rewinding to shortly before the detection and sending very slightly delayed data. This kind of test requires a certain number of listeners to judge recordings containing the processing results of the VADs being tested, giving marks to several speech sequences on the following features: Quality; Comprehension difficulty; Audibility of clipping. These marks are then used to calculate average results for each of the features listed above, thus providing a global estimate of the behavior of the VAD being tested. To conclude, whereas objective methods are very useful in an initial stage to evaluate the quality of a VAD, subjective methods are more significant. As they require the participation of several people for a few days, increasing cost, they are generally only used when a proposal is about to be standardized. == Implementations == One early standard VAD is that developed by British Telecom for use in the Pan-European digital cellular mobile telephone service in 1991. It uses inverse filtering trained on non-speech segments to filter out background noise, so that it can then more reliably use a simple power-threshold to decide if a voice is present. The G.729 standard calculates the following features for its VAD: line spectral frequencies, full-band energy, low-band energy (<1 kHz), and zero-crossing rate. It applies a simple classification using a fixed decision boundary in the space defined by these features, and then applies smoothing and adaptive correction to improve the estimate. The GSM standard includes two VAD options developed by ETSI. Option 1 computes the SNR in nine bands and applies a threshold to these values. Option 2 calculates different parameters: channel power, voice metrics, and noise power. It then thresholds the voice metrics using a threshold that varies according to the estimated SNR. The Speex audio compression library uses a procedure named Improved Minima Controlled Recursive Averaging, which uses a smoothed representation of spectral pow
Conference on Neural Information Processing Systems
The Conference on Neural Information Processing Systems (abbreviated as NeurIPS and formerly NIPS) is a machine learning and computational neuroscience conference held annually in December. Along with ICLR and ICML, it is one of the three primary conferences of high impact in machine learning and artificial intelligence research. The conference includes three days of invited talks along with oral and poster presentations of refereed papers, followed by two days of workshops and competitions. == History == The NeurIPS meeting was first proposed in 1986 at the annual invitation-only Snowbird Meeting on Neural Networks for Computing organized by The California Institute of Technology and Bell Laboratories. NeurIPS was designed as a complementary open interdisciplinary meeting for researchers exploring biological and artificial Neural Networks. Reflecting this multidisciplinary approach, NeurIPS began in 1987 with information theorist Ed Posner as the conference president and learning theorist Yaser Abu-Mostafa as program chairman. Research presented in the early NeurIPS meetings included a wide range of topics from efforts to solve purely engineering problems to the use of computer models as a tool for understanding biological nervous systems. Since then, the biological and artificial systems research streams have diverged, and recent NeurIPS proceedings have been dominated by papers on machine learning, artificial intelligence and statistics. From 1987 until 2000 NeurIPS was held in Denver, United States. Since then, the conference was held in Vancouver, Canada (2001–2010), Granada, Spain (2011), and Lake Tahoe, United States (2012–2013). In 2014 and 2015, the conference was held in Montreal, Canada, in Barcelona, Spain in 2016, in Long Beach, United States in 2017, in Montreal, Canada in 2018 and Vancouver, Canada in 2019. Reflecting its origins at Snowbird, Utah, the meeting was accompanied by workshops organized at a nearby ski resort up until 2013, when it outgrew ski resorts. The first NeurIPS Conference was sponsored by the IEEE. The following NeurIPS Conferences have been organized by the NeurIPS Foundation, established by Ed Posner. Terrence Sejnowski has been the president of the NeurIPS Foundation since Posner's death in 1993. The board of trustees consists of previous general chairs of the NeurIPS Conference. The first proceedings was published in book form by the American Institute of Physics in 1987, and was entitled Neural Information Processing Systems, then the proceedings from the following conferences have been published by Morgan Kaufmann (1988–1993), MIT Press (1994–2004) and Curran Associates (2005–present) under the name Advances in Neural Information Processing Systems. The conference was originally abbreviated as "NIPS". By 2018 a few commentators were criticizing the abbreviation as encouraging sexism due to its association with the word nipples, and as being a slur against Japanese. The board changed the abbreviation to "NeurIPS" in November 2018. == Topics == Along with machine learning and neuroscience, other fields represented at NeurIPS include cognitive science, psychology, computer vision, statistical linguistics, and information theory. Over the years, NeurIPS became a premier conference on machine learning and although the 'Neural' in the NeurIPS acronym had become something of a historical relic, the resurgence of deep learning in neural networks since 2012, fueled by faster computers and big data, has led to achievements in speech recognition, object recognition in images, image captioning, language translation and world championship performance in the game of Go, based on neural architectures inspired by the hierarchy of areas in the visual cortex (ConvNet) and reinforcement learning inspired by the basal ganglia (Temporal difference learning). Notable affinity groups have emerged from the NeurIPS conference and displayed diversity, including Black in AI (in 2017), Queer in AI (in 2016), and others. === Named lectures === In addition to invited talks and symposia, NeurIPS also organizes two named lectureships to recognize distinguished researchers. The NeurIPS Board introduced the Posner Lectureship in honor of NeurIPS founder Ed Posner; two Posner Lectures were given each year up to 2015. Past lecturers have included: 2010 – Josh Tenenbaum and Michael I. Jordan 2011 – Rich Sutton and Bernhard Schölkopf 2012 – Thomas Dietterich and Terry Sejnowski 2013 – Daphne Koller and Peter Dayan 2014 – Michael Kearns and John Hopfield 2015 – Zoubin Ghahramani and Vladimir Vapnik 2016 – Yann LeCun 2017 – John Platt 2018 – Joëlle Pineau 2019 – Yoshua Bengio 2020 – Christopher Bishop 2021 – Peter Bartlett In 2015, the NeurIPS Board introduced the Breiman Lectureship to highlight work in statistics relevant to conference topics. The lectureship was named for statistician Leo Breiman, who served on the NeurIPS Board from 1994 to 2005. Past lecturers have included: 2015 – Robert Tibshirani 2016 – Susan Holmes 2017 – Yee Whye Teh 2018 – David Spiegelhalter 2019 – Bin Yu 2020 – Marloes Maathuis 2021 – Gabor Lugosi 2022 – Emmanuel Candes 2023 – Susan Murphy 2024 – Arnaud Doucet == NeurIPS consistency experiment == In NIPS 2014, the program chairs duplicated 10% of all submissions and sent them through separate reviewers to evaluate randomness in the reviewing process. Several researchers interpreted the result. Regarding whether the decision in NIPS is completely random or not, John Langford writes: "Clearly not—a purely random decision would have arbitrariness of ~78%. It is, however, quite notable that 60% is much closer to 78% than 0%." He concludes that the result of the reviewing process is mostly arbitrary. In NeurIPS 2021, the program chairs repeated the 2014 experiment and found similar levels of review inconsistency; 23% of duplicated submissions received different accept/reject decisions, and 50.6% of accepted papers would have been rejected under re-review. == Locations == 1987–2000: Denver, Colorado, United States 2001–2010: Vancouver, British Columbia, Canada 2011: Granada, Spain 2012 & 2013: Stateline, Nevada, United States 2014 & 2015: Montréal, Quebec, Canada 2016: Barcelona, Spain 2017: Long Beach, California, United States 2018: Montréal, Quebec, Canada 2019: Vancouver, British Columbia, Canada 2020: Vancouver, British Columbia, Canada (virtual conference) 2021: Virtual conference 2022 & 2023: New Orleans, Louisiana, United States 2024: Vancouver, British Columbia, Canada 2025: San Diego, California, United States and Mexico City, Mexico 2026: Sydney, New South Wales, Australia, with satellite events in Atlanta and Paris