Hallucination improvements in Gemini 4 Hydrogen: cautions and remedies for PMIDs and DOIs

Contents
  1. About Gemini 4 Hydrogen
  2. Results of asking for source links
  3. Asking for PMIDs or DOIs, and using the Part 1 prompt
  4. Asking with the /codex-search prompt
  5. The Part 2 Gem and the migration to skills
  6. Organize the question before searching
  7. The purpose of the search and what to check
  8. Recommended way to ask and to check
  9. What I tested
  10. Conditions and results
  11. References

At the time of Part 11, in November 2025, asking Gemini to "include links to your sources" sometimes made it answer from memory without searching. The titles and authors were right, but the URLs did not exist. If the linked page does not exist, neither the recommendation nor the numbers can be checked. In Part 1, I therefore introduced an English prompt that asks Gemini to search and to check bibliographic details. In Part 22, published in January 2026, I released a Gem, "Clinical Evidence Mapper", with detailed instructions to answer only from search results.

I made the same request as in Part 1, "Please include links to your sources", with Gemini 4 Hydrogen3, released in November 2026. Across three clinical questions asked twice each, Gemini searched in all six runs, and all the papers and pages behind the 24 links (counting duplicates) existed. I used neither a long prompt nor a Gem. However, in the two runs where I also asked for PMIDs or DOIs, one run included PMIDs that pointed to other papers. In the six runs where I asked for guideline recommendations and absolute effects, all the guidelines cited existed, but in some runs the strength of a recommendation or a number differed from the source. It is not perfect yet, but it is becoming usable as a tool for finding literature.

Key points: results by way of asking
How I askedResult
"Please include links to your sources" (6 runs)All 6 runs searched, and all 24 links led to sources that existNo long prompt or Gem was used
Also asking for PMIDs or DOIs (2 runs)In 1 run, 3 of 6 papers had PMIDs that pointed to other papersCheck the numbers by searching PubMed for the title
The /codex-search prompt (3 runs)All 15 DOIs and 6 PMIDs pointed to the correct papersOne run per question
Asking for guideline recommendations and absolute effects (6 runs)All 23 cited guidelines existIn 3 runs, the strength or target population of a recommendation differed from the guideline text, and some numbers did not match the sources

Tested by the author in the Gemini app, November 15 and 16, 2026

About Gemini 4 Hydrogen

Gemini 4 Hydrogen is a model that Google released on November 12, 2026 (US time)3. It can be chosen in the model picker of the Gemini app, and in Google Search it is provided to all users, so it can be used free of charge3.

According to Google, it can take in PDFs, spreadsheets, charts, audio, and recorded video and turn them into cited reports3. Google also states that it cut fabricated claims by more than 5x compared with Gemini 3.8 Flash3.

Features of Gemini 4 Hydrogen as described by Google
ItemWhat Google says
ReleaseNovember 12, 2026 (US time)Announced on Google's official blog
Where to use itModel picker in the Gemini app, Google SearchProvided to all users in Google Search, so it can be used free of charge
InputsPDFs, spreadsheets, charts, audio, recorded videoTurned into cited reports
HallucinationsCut by more than 5x compared with Gemini 3.8 FlashGoogle's own evaluation. Google's term is "fabricated claims"

Kavukcuoglu K. Introducing Gemini 4 Hydrogen3

On X, a post4 showing how to have Gemini in Chrome summarize the YouTube video open in the browser received about 1,000 likes (as of November 16, 2026).

The reduction in hallucinations is a figure from Google's own evaluation. For the sources of medical literature, I tested three clinical questions.

A person at a desk working on a laptop. Three cards float beside the laptop, connected by lines, and one has a pink bookmark
After your question, ask for links to sources

Results of asking for source links

I tested three clinical questions: exercise therapy for depression, cognitive behavioral therapy for insomnia in older adults, and SGLT2 inhibitors in type 2 diabetes. For example, I entered "In adults with depression, does exercise therapy improve depressive symptoms? Please include links to your sources." The questions were asked in Japanese.

Results for the same request, "Please include links to your sources"
November 2025 (Part 1)
Gemini sometimes answered from memory without searching. Titles and authors were correct, but the URLs did not exist
November 2026 (Gemini 4 Hydrogen)
Gemini searched in all six runs. All papers and pages behind 24 links (counting duplicates) existed

In all six runs, the answer showed sources from search results. Two of the 24 links (counting duplicates) used an old URL of a society journal that did not open, and I confirmed those papers by DOI.

In the first run of the depression question, the Cochrane review updated in January 2026 (73 randomized controlled trials, at least 4,985 participants)5 was cited. The numbers of trials and participants in the answer agreed with the review abstract.

The sources included papers and systematic reviews as well as society and Ministry of Health, Labour and Welfare pages, a Japanese Cochrane summary, and a university blog. No PMIDs or DOIs were given, and the sources were not formatted as a reference list. When you want to check a society recommendation or a national guideline, society and government pages are a good starting point.

Asking for PMIDs or DOIs, and using the Part 1 prompt

For cases where the answer is to be used as a list of papers, I also asked for PMIDs or DOIs. When I wrote "Prioritize original studies, systematic reviews, and clinical practice guidelines, and include the PMID or DOI where you can confirm it" after "Please include links to your sources", the PMIDs and DOIs of all 5 papers were correct for the depression question. For the insomnia question in older adults, the PMIDs of 3 of 6 papers pointed to other papers. Opening the PubMed links in the answer showed unrelated papers, such as a review of immunotherapy for tuberculosis. In addition, one URL presented as a Japan Geriatrics Society page led to a nonexistent page.

Results of the two runs that also asked for PMIDs or DOIs
DepressionAll 5 correctInsomnia in older adults3 pointed to other papersPMID correctPMID pointed to another paper
Paper cited in the answerWhere the given PMID led
Morin 1999JAMAA different CDC articleDOI was correct
Montgomery 2003CochraneA review of immunotherapy for tuberculosisDOI was correct
Rios 2019BMJ OpenAn unrelated paperThe DOI also led to an unrelated paper. A paper by the same author in the same year is in a different journal (Systematic Reviews)

With the English prompt from Part 1, the list consisted mainly of peer-reviewed papers with PMIDs. There was no case in which a PMID pointed to another paper. However, in one of seven runs, Gemini did not search. In that run, it noted, in effect, that it could not access external resources and would list only references whose bibliographic details were established, and the newest reference was from 2021. Correct bibliographic details do not mean that recent studies were checked. When I sent the same question again, it searched and also cited the 2023 European insomnia guideline6.

In both cases the number of runs was small, and it is unknown why the wrong PMIDs and the run without a search occurred.

Asking with the /codex-search prompt

/codex-search7 is a skill that has Codex CLI run a web search, and I have published it on GitHub. Its prompt requires a DOI for every paper, "DOI: none" when no DOI is found, and "unknown" for anything that cannot be confirmed. The same prompt can be sent to Gemini 4 Hydrogen as it is. I asked the same three clinical questions once each with Gemini and with /codex-search.

To use it in Gemini, paste the prompt below into a regular chat, replace the bracketed text after "Search query:" with your question, and send it. The test used the original Japanese prompt; the English version below is a translation.

The /codex-search prompt (also usable in Gemini)
Search the web and collect detailed information on the following query.

Search query: (write your question here)

Requirements:
- Prefer primary sources such as official documentation, official blogs, specifications, and GitHub
- Do not fabricate numbers, specifications, or limits. Write "unknown" if unclear
- State the date the information was accessed
- Output the search results in structured Markdown
- Include these headings in the output:
  - Summary (1-3 sentence summary)
  - Key Findings (bullet points)
  - Sources (URL list)
  - Details (detailed findings)

[Important] For literature searches (papers and academic articles):
- For every paper or article found, always extract and state its DOI (Digital Object Identifier)
- A DOI usually has the form "10.xxxx/..."
- If no DOI is found, state "DOI: none"
- Example output format:
  ## Paper 1
  - Title: xxx
  - Authors: xxx
  - DOI: 10.xxxx/xxx
  - URL: https://doi.org/10.xxxx/xxx

Source: scripts/codex_search.sh in codex-search7. Translated from Japanese; only the part after "Search query:" was changed

Results with the same prompt (three questions, one run each)
ItemGemini 4 Hydrogen/codex-search
SearchSearched in all 3 runsSearched in all 3 runs
References with a DOI15All pointed to the correct paper22All pointed to the correct paper
"DOI: none"1A paper in the Japanese Journal of Public Health1The VA/DoD insomnia guideline
PMIDs6All correct3All correct
Time per questionAbout 1 minuteAbout 2 to 3 minutes

Conducted by the author on November 16, 2026. Gemini at thinking level "Low"; /codex-search with Codex CLI (gpt-6-sol, reasoning effort medium)

Morin 1999, whose PMID had pointed to another paper with the shorter request that also asked for PMIDs or DOIs, came back with the correct PMID and DOI under this prompt. /codex-search took longer, but it listed more references, including many recent ones such as clinical practice guidelines from 2026. Each was run only once, and it is unknown whether the difference in prompts explains the result.

I also pasted the instructions of the Part 2 Gem, "Clinical Evidence Mapper"2, into a regular chat and asked the same three questions once each. Gemini searched in only one run, the depression question. In the other two runs, it stated that it had not run the web search tool and listed no references. No fictitious reference appeared. In the run with a search, all 5 references existed. However, the body text gave a co-author, Parker, as the first author of the BMJ paper, and the reference list did not state PMID or DOI numbers.

The Part 2 Gem and the migration to skills

For personal Google accounts of users aged 18 and over, Gems will automatically migrate to skills on November 17, 2026, and will be removed from the Gemini app on December 178. The Gem released in Part 2, "Clinical Evidence Mapper", will no longer be available.

Organize the question before searching

Checking that the linked pages exist does not tell you whether the answer can be used in practice. So what should you look for in the first place, and which parts of the answer should you check?

The starting point is to organize the question yourself before asking. Break the question from clinical practice into four parts: population (P), intervention (I), comparison (C), and outcome (O). This format is called PICO. Once the four parts are set, they determine which literature to look for and which points to check in the answer.

Organizing a clinical question with PICO (example: insomnia in older adults)
P: Population
Older adults with insomnia
I: Intervention
Cognitive behavioral therapy for insomnia (CBT-I)
C: Comparison
Sleep medication
O: Outcome
Whether insomnia symptoms improve

In evidence-based medicine (EBM), checking the recommendations of trustworthy clinical practice guidelines is considered central to clinical decisions such as choosing a treatment9. Knowing the size of the benefits and harms as absolute effects, and the certainty of those estimates, is also considered important1011. Even for the same question, which parts of the answer have to be correct depends on why you are asking.

Three things to check as the evidence for a clinical decision
Guideline recommendation
What a trustworthy guideline recommends, for whom, and how strongly
Size of benefits and harms
Read as absolute effects: for how many people out of 1,000 the outcome changes
Certainty
How certain the estimate is. GRADE rates it at four levels: high, moderate, low, and very low

The purpose of the search and what to check

What matters for each purpose, and how to ask and check
PurposeWhat matters, and how to ask and check
Finding papers or pages to readThe linked pages existAsk "Please include links to your sources". Open each link and check whether the content matches your question
Building a reference list for a paper or grant applicationThe PMIDs and DOIs are correctAlso ask for PMIDs or DOIs. Check each number by searching PubMed for the title
Finding the evidence for a clinical decision, such as a treatment choiceGuideline recommendations, the size of benefits and harms, and their certaintyAsk "Show the recommendations of trustworthy clinical practice guidelines, the size of the benefits and harms (absolute effects), and their certainty. Please include links to your sources". Check recommendations and numbers in the text of the linked guideline

In the six runs that asked for the evidence for a clinical decision, every answer gave all three items: recommendations, absolute effects, and certainty. The direction of the recommendation, for or against, agreed with the guideline text in all 22 recommendations I checked. The differences were in the strength and target population of recommendations, whether a revision was reflected, and the numbers. For the type 2 diabetes question, both runs cited the 2021 BMJ clinical practice guideline12. That guideline makes a strong recommendation for SGLT2 inhibitors only for people who have both cardiovascular disease and chronic kidney disease, and a weak recommendation for people who have only one of them. One of the answers stated a strong recommendation for people with either one. The NICE guideline NG2813 in the UK was revised in 2026, but the answer that cited it described the content before the revision. The absolute-effect numbers came from different sources in different runs of the same question, and the values differed. If the AI's answer states a recommendation or a number, check it in the text of the linked guideline, in this order: the year of revision, the target population and strength of the recommendation, and the numbers.

Recommended way to ask and to check

Steps to check sources with Gemini
  1. Write "Please include links to your sources" after your questionAsk this way first. You do not need to prepare a long prompt from the start.
  2. Open the links behind any conclusion you citeCheck not only the title and year but also whether the study population, intervention and comparison, and outcomes match your question.
  3. Check PMIDs and DOIs by titleSearch PubMed for the title. Numbers of other papers can be mixed in.
A person at a desk holding a smartphone in one hand and turning the pages of a paper with the other to compare them. A pink clip is attached to the paper
Open the links in the answer and compare them with the papers

When you want a list of papers with DOIs, the /codex-search prompt7 is another option.

The sources section may include pages related to only part of the answer14. Check important conclusions at the linked pages before using them. If an answer says something like "cannot search," send the question again or search PubMed directly.

Even when an AI cites sources that exist, it is the user who checks them against the original sources. Before asking, think about which guideline you would look for and which numbers you would check; this makes errors in the answer easier to notice. While using AI, I think it is important to keep building our own ability to think these questions through.

Take-home message
How to ask
When "Please include links to your sources" was added after the question, Hydrogen searched and cited sources that exist in all 6 runs
How to check
Check PMIDs and DOIs by searching PubMed for the title. Check the strength and target of recommendations, and the numbers, in the guideline text
Before asking
Organize the question with PICO, and decide which guideline to look for and which numbers to check

What I tested

I asked three questions on topics with many systematic reviews on November 15 and 16, 2026. Each way of asking was tested in two to seven runs. I selected Hydrogen and used the default thinking level, "Low". Without a Google AI plan, most prompts are answered by the lightweight Flash-Lite15.

Conditions and results

In the Gemini app, I set the model to "4 Hydrogen" and tested the following six conditions.

  • Links only: write "Please include links to your sources" after the question. Thinking level "Low". Two runs for each of the three questions.
  • Links plus PMID or DOI: write "Prioritize original studies, systematic reviews, and clinical practice guidelines, and include the PMID or DOI where you can confirm it" after "Please include links to your sources". Thinking level "Low". One run each for the depression and insomnia questions.
  • Part 1 prompt: add the English prompt from Part 1 after the question. Thinking level "Low" in three runs and "High" in four runs. The one run without a search was at "High".
  • /codex-search prompt: put the question in the "search query" line of the prompt and send it. Thinking level "Low". One run for each of the three questions (November 16).
  • Part 2 Gem instructions: paste the instructions into a regular chat, write "Clinical question:" and the question at the end, and send it. Thinking level "Low". One run for each of the three questions (November 16).
  • Evidence for a clinical decision: write "Show the recommendations of trustworthy clinical practice guidelines, the size of the benefits and harms (absolute effects), and their certainty. Please include links to your sources" after the question. Thinking level "Low". Two runs for each of the three questions (November 16).

For the returned sources, I checked papers against PubMed, Crossref, and DOI for title, first author, and year, and opened society and government pages to confirm their titles. In the runs that asked for the evidence for a clinical decision, I opened the guidelines cited to confirm their titles and compared the recommendations and numbers with the guideline text and the underlying systematic reviews. In those runs, I confirmed whether Gemini had searched from the "thinking process" display of the answer.

Results of asking Gemini 4 Hydrogen for sources
ConditionResult
Links only Searched in 6 of 6 runsAll papers and pages pointed to by 24 links (counting duplicates) existed. Two links used an old URL of a society journal that did not open, and were confirmed by DOI
Links plus PMID or DOI Searched in both runsIn the first run, the PMIDs of all 5 papers were correct. In the second run, the PMIDs of 3 of 6 papers pointed to other papers, and one more URL led to a nonexistent page
Part 1 prompt Searched in 6 of 7 runsAll 45 references existed. Bibliographic details matched for 44 (one with an abbreviated title), and one differed only in how the publication year was given
/codex-search prompt Searched in all 3 runsAll 15 references with a DOI and all 6 PMIDs were correct. One reference was marked "DOI: none"
Part 2 Gem instructions Searched in 1 of 3 runsIn the run with a search, all 5 references existed. The body text gave a co-author as the first author of the BMJ paper, and the reference list did not state PMID or DOI numbers. In the 2 runs without a search, Gemini listed no references and stated that it had not searched
Evidence for a clinical decision Searched in 6 of 6 runsAll 23 guidelines cited (counting duplicates) existed. The link led to that guideline for 21; one link did not open and one led to a different document. The direction of the recommendation agreed with the text in all 22 checked. Strength, target population, or details differed in three runs (five recommendations). For the numbers, all those that could be checked against the source agreed in the two insomnia runs; in the other four runs, some disagreed or could not be confirmed. All 10 PMIDs and PMCIDs (counting duplicates) pointed to the correct paper

Conducted by the author in the Gemini app on November 15 and 16, 2026

References

  1. Koda M. How to include references in Gemini. 2025. /en/articles/2898/
  2. Koda M. How to add references in Gemini (Part 2): Gem and prompt released. 2026. /en/articles/142801125/
  3. Kavukcuoglu K. Introducing Gemini 4 Hydrogen. Google. 2026. https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-hydrogen/
  4. @studio_yebisu. Post on X (Japanese). 2026. (Accessed November 16, 2026) https://x.com/studio_yebisu/status/2121835594674561098
  5. Clegg AJ, et al. Exercise for depression. Cochrane Database Syst Rev. 2026;1(1):CD004366. https://doi.org/10.1002/14651858.CD004366.pub7
  6. Riemann D, et al. The European Insomnia Guideline: An update on the diagnosis and treatment of insomnia 2023. J Sleep Res. 2023;32(6):e14035. https://doi.org/10.1111/jsr.14035
  7. Koda M. codex-search. GitHub. 2026. (Accessed November 16, 2026) https://github.com/shoei05/codex-search
  8. Gemini Apps Help. About migrating Gems to skills (Japanese). (Accessed November 15, 2026) https://support.google.com/gemini/answer/18560919?hl=ja
  9. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390:415-423. https://doi.org/10.1016/S0140-6736(16)31592-6
  10. Guyatt GH, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924-926. https://doi.org/10.1136/bmj.39489.470347.AD
  11. Guyatt GH, et al. GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes. J Clin Epidemiol. 2013;66:158-172. https://doi.org/10.1016/j.jclinepi.2012.01.012
  12. Li S, et al. SGLT-2 inhibitors or GLP-1 receptor agonists for adults with type 2 diabetes: a clinical practice guideline. BMJ. 2021;373:n1091. https://doi.org/10.1136/bmj.n1091
  13. National Institute for Health and Care Excellence. Type 2 diabetes in adults: management. NICE guideline NG28. (Accessed November 16, 2026) https://www.nice.org.uk/guidance/ng28
  14. Gemini Apps Help. View related sources in Gemini Apps (Japanese). (Accessed November 15, 2026) https://support.google.com/gemini/answer/14143489?hl=ja
  15. Gemini Apps Help. About Gemini models. (Accessed November 15, 2026) https://support.google.com/gemini/answer/18648722?hl=en
English translation of the Japanese article.