Gradio

Welcome to the Leaderboard for open Hebrew LLMs. The leaderboard ranks the different models according to their success on various tasks on Hebrew.

The leaderboard was created and is operated by a collaboration of Mafat / The Israeli National Program for NLP in Hebrew and Arabic and DICTA: The Israel Center for Text Analysis.

ברוכים הבאים ללוח התוצאות של מודלי LLM פתוחים בעברית. לוח התוצאות מדרג את המודלים השונים לפי הצלחתם במשימות שונות בעברית.

לוח התוצאות נוצר ומתופעל על ידי שיתוף פעולה בין מפא"ת / התוכנית הלאומית הישראלית ל-NLP בעברית ובערבית ודיקטה: המרכז הישראלי לניתוח טקסטים


⭕	mistralai/Mistral-Large-Instruct-2407	75.71	90.95	87.16	75.5	88.49	34.73	77.41	122.61	mistralai/Mistral-Large-Instruct-2407
⭕	ai21labs/AI21-Jamba-Large-1.6	73.42	93.1	81.5	68.2	81.65	35.98	80.07	398.56	ai21labs/AI21-Jamba-Large-1.6
🟢	meta-llama/Meta-Llama-3.1-70B	72.57	88.57	80.31	77.5	80.94	34.7	73.42	70.55	meta-llama/Meta-Llama-3.1-70B
⭕	CohereForAI/c4ai-command-r-plus-08-2024	72.36	91.67	82.45	67.83	85.25	36.85	70.1	103.81	CohereForAI/c4ai-command-r-plus-08-2024
⭕	CohereForAI/c4ai-command-r-plus	71.76	91.67	82.71	65.9	84.53	36	69.77	103.81	CohereForAI/c4ai-command-r-plus
🟢	meta-llama/Meta-Llama-3-70B	71.36	87.86	78.38	76.7	81.29	34.47	69.44	70.55	meta-llama/Meta-Llama-3-70B
🟦	meta-llama/Llama-3.3-70B-Instruct	70.53	77.14	84.38	77.33	81.65	33.91	68.77	70.55	meta-llama/Llama-3.3-70B-Instruct
⭕	CohereForAI/aya-expanse-32b	70.24	92.14	83.42	67.97	82.73	34.7	60.47	32.3	CohereForAI/aya-expanse-32b
🟢	Qwen/Qwen2.5-72B	70.18	95.48	87	71.5	74.82	32.79	59.47	72.71	Qwen/Qwen2.5-72B
⭕	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF	69.92	77.86	82.51	75.5	83.81	33.39	66.45	70.55	nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
⭕	meta-llama/Llama-3.1-70B-Instruct	69.54	74.52	83.88	75.53	83.81	33.71	65.78	70.55	meta-llama/Llama-3.1-70B-Instruct
🟢	google/gemma-3-27b-pt	69.5	85.24	78.27	64.97	81.65	36.45	70.43	27.43	google/gemma-3-27b-pt
🟢	meta-llama/Llama-4-Scout-17B-16E	68.71	90	78.81	67.83	79.14	35.34	61.13	108.64	meta-llama/Llama-4-Scout-17B-16E
⭕	CohereForAI/c4ai-command-r-08-2024	67.71	79.76	76.19	67	82.37	35.81	65.12	32.3	CohereForAI/c4ai-command-r-08-2024
⭕	mistralai/Mistral-Small-24B-Instruct-2501	67.56	83.1	85.97	71.67	79.5	29.95	55.15	23.57	mistralai/Mistral-Small-24B-Instruct-2501
🟦	Qwen/QwQ-32B-Preview	66.04	87.38	83.62	73.87	77.34	29.49	44.52	32.76	Qwen/QwQ-32B-Preview
🟢	mistralai/Mistral-Small-3.1-24B-Base-2503	66	81.19	77.86	66.73	80.94	30.81	58.47	24.01	mistralai/Mistral-Small-3.1-24B-Base-2503
⭕	nvidia/Llama-3_3-Nemotron-Super-49B-v1_5	65.92	80.71	83.29	74.4	76.62	29.65	50.83	49.9	nvidia/Llama-3_3-Nemotron-Super-49B-v1_5
🟢	nvidia/Nemotron-H-56B-Base-8K	65.7	83.57	81.92	69.77	77.34	29.1	52.49	56.3	nvidia/Nemotron-H-56B-Base-8K
⭕	CohereForAI/aya-23-35B	65.42	87.86	79.44	62.33	80.58	33.15	49.17	34.98	CohereForAI/aya-23-35B
🔶	618AI/dictalm2-it-qa-fine-tune	64.78	75.95	74.09	61.53	71.94	35.37	69.77	7.25	618AI/dictalm2-it-qa-fine-tune
🟢	mistralai/Mistral-Small-24B-Base-2501	64.28	83.1	75.01	64.8	80.22	30.74	51.83	23.57	mistralai/Mistral-Small-24B-Base-2501
⭕	rombodawg/Rombos-LLM-V2.5-Qwen-32b	64.06	90.24	71.62	77.27	76.26	28.11	40.86	32.76	rombodawg/Rombos-LLM-V2.5-Qwen-32b
🟢	google/gemma-3-12b-pt	64.04	85.71	71.47	59.6	74.82	34.47	58.14	12.19	google/gemma-3-12b-pt
🟦	google/gemma-2-9b-it	63.76	87.62	83.72	70.33	72.66	30.05	38.21	9.24	google/gemma-2-9b-it
⭕	Qwen/Qwen3-32B	63.58	88.57	83.51	71.73	67.99	29.13	40.53	32.76	Qwen/Qwen3-32B
🟦	UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3	63.2	88.57	79.66	70.7	71.22	29.85	39.2	9.24	UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3
🟢	Qwen/Qwen2.5-14B	63.13	90	81.13	71.73	68.35	28.71	38.87	14.77	Qwen/Qwen2.5-14B
🟢	Qwen/Qwen3-14B-Base	63.04	84.76	80.88	70.2	73.74	30.12	38.54	14.77	Qwen/Qwen3-14B-Base
⭕	Qwen/Qwen3-30B-A3B-Instruct-2507	63	90.48	77.38	71.37	69.42	27.83	41.53	30.53	Qwen/Qwen3-30B-A3B-Instruct-2507
⭕	ai21labs/AI21-Jamba-Mini-1.6	62.93	72.62	75.74	61.93	72.66	31.86	62.79	51.57	ai21labs/AI21-Jamba-Mini-1.6
⭕	Qwen/Qwen3-30B-A3B-Thinking-2507	62.84	89.05	76.08	72.07	70.14	28.52	41.2	30.53	Qwen/Qwen3-30B-A3B-Thinking-2507
⭕	ai21labs/AI21-Jamba-1.5-Large	62.73	83.1	66.15	60.7	81.65	12.36	72.43	398.56	ai21labs/AI21-Jamba-1.5-Large
🟢	dicta-il/dictalm2.0	62.57	79.05	67.8	59.67	66.91	35.57	66.45	7.25	dicta-il/dictalm2.0
🟦	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	62.34	85.71	76.9	74	70.86	27.69	38.87	32.76	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
🔶	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	62.27	85.95	76.9	73.7	70.5	27.71	38.87	32.76	deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
🟢	Qwen/Qwen3-30B-A3B-Base	61.85	93.33	73	64.63	68.71	29.26	42.19	30.53	Qwen/Qwen3-30B-A3B-Base
🟢	microsoft/phi-4	61.53	76.9	74.71	69.63	75.54	29.51	42.86	14.66	microsoft/phi-4
⭕	dicta-il/dictalm2.0-instruct	61.45	73.57	67.98	56.3	69.42	35.3	66.11	7.25	dicta-il/dictalm2.0-instruct
⭕	Qwen/Qwen3-14B	61.43	83.57	82.07	71.67	66.55	27.87	36.88	14.77	Qwen/Qwen3-14B
🟢	Qwen/Qwen2.5-32B	61.11	90	86.49	74.07	42.09	30.81	43.19	32.76	Qwen/Qwen2.5-32B
⭕	CohereForAI/c4ai-command-r-v01	61.02	73.57	69.13	59.07	75.54	30.99	57.81	34.98	CohereForAI/c4ai-command-r-v01
⭕	CohereLabs/c4ai-command-r7b-arabic-02-2025	60.16	83.57	71.8	61.63	71.94	32.15	39.87	8.03	CohereLabs/c4ai-command-r7b-arabic-02-2025
🔶	SicariusSicariiStuff/Impish_Nemo_12B	59.97	73.1	69.71	63.7	78.42	28.39	46.51	0	SicariusSicariiStuff/Impish_Nemo_12B
⭕	ronigold/dictalm2.0-instruct-fine-tuned-alpaca-gpt4-hebrew	59.36	67.38	65.1	59.97	65.11	33.51	65.12	7.25	ronigold/dictalm2.0-instruct-fine-tuned-alpaca-gpt4-hebrew
🟢	Qwen/Qwen3-8B-Base	59.36	75.71	78.94	70.4	67.27	27.64	36.21	8.19	Qwen/Qwen3-8B-Base
⭕	SicariusSicariiStuff/Zion_Alpha_Instruction_Tuned	59.36	86.43	71.36	64.1	70.14	30.9	33.22	7.24	SicariusSicariiStuff/Zion_Alpha_Instruction_Tuned
⭕	mistralai/Mistral-Nemo-Instruct-2407	59.19	72.38	73.4	65.77	72.66	27.41	43.52	12.25	mistralai/Mistral-Nemo-Instruct-2407
🔶	SeaLLMs/SeaLLM-7B-v2.5	58.99	79.05	71.97	67.03	70.86	27.17	37.87	8.54	SeaLLMs/SeaLLM-7B-v2.5
🟢	google/gemma-2-9b	58.82	73.57	69.24	65.23	70.86	31.49	42.52	9.24	google/gemma-2-9b
⭕	CohereForAI/aya-expanse-8b	58.49	77.14	69.35	63.4	71.58	31.24	38.21	8.03	CohereForAI/aya-expanse-8b
⭕	Qwen/Qwen3-8B	57.63	80	78.53	68.3	63.31	25.73	29.9	8.19	Qwen/Qwen3-8B
🔶	SicariusSicariiStuff/Zion_Alpha	57.61	84.05	67.67	65.67	65.83	27.93	34.55	7.24	SicariusSicariiStuff/Zion_Alpha
🔶	deepseek-ai/DeepSeek-R1-Distill-Qwen-14B	57.6	85.48	71.38	69.2	63.67	22.99	32.89	14.77	deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
⭕	SicariusSicariiStuff/Zion_Alpha_Instruction_Tuned_SLERP	57.37	78.1	73	70.3	61.87	27.7	33.22	7.24	SicariusSicariiStuff/Zion_Alpha_Instruction_Tuned_SLERP
⭕	ai21labs/AI21-Jamba-1.5-Mini	57.04	69.52	69.38	50.5	73.02	22	57.81	51.57	ai21labs/AI21-Jamba-1.5-Mini
🟢	google/gemma-2-27b	56.52	63.81	74.22	58.73	75.54	17.99	48.84	27.23	google/gemma-2-27b
🟢	Qwen/Qwen2.5-7B	56.3	81.9	67.51	66.73	62.59	25.5	33.55	7.62	Qwen/Qwen2.5-7B
🟢	mistralai/Mistral-Nemo-Base-2407	55.61	65.95	68.48	57.47	71.22	28.99	41.53	12.25	mistralai/Mistral-Nemo-Base-2407
🔶	ronigold/dictalm2.0-instruct-fine-tuned	55.57	55.74	65.15	49.17	64.75	33.95	64.67	7.25	ronigold/dictalm2.0-instruct-fine-tuned
🟦	mlabonne/NeuralDaredevil-8B-abliterated	55.21	78.81	66.27	57.57	66.55	26.2	35.88	8.03	mlabonne/NeuralDaredevil-8B-abliterated
🔶	mistralai/Codestral-22B-v0.1	54.92	74.76	70.97	64.8	64.75	24.65	29.57	22.25	mistralai/Codestral-22B-v0.1
🟦	meta-llama/Meta-Llama-3.1-8B-Instruct	54.83	73.81	70.47	57.87	69.42	25.86	31.56	8.03	meta-llama/Meta-Llama-3.1-8B-Instruct
🔶	NousResearch/Hermes-3-Llama-3.1-8B	54.55	64.52	65.52	66.77	67.63	26.68	36.21	8.03	NousResearch/Hermes-3-Llama-3.1-8B
🟦	vicgalle/Configurable-Hermes-2-Pro-Llama-3-8B	54.13	79.29	61.99	63.73	62.95	25.56	31.23	8.03	vicgalle/Configurable-Hermes-2-Pro-Llama-3-8B
⭕	CohereForAI/aya-23-8B	54.06	65.71	66.41	59.23	66.55	30.26	36.21	8.03	CohereForAI/aya-23-8B
⭕	mistralai/Mistral-Small-Instruct-2409	53.83	63.33	76.58	62.57	64.75	22.86	32.89	22.25	mistralai/Mistral-Small-Instruct-2409
🟢	Qwen/Qwen1.5-32B	53.81	67.86	68.85	66.93	61.15	26.52	31.56	32.51	Qwen/Qwen1.5-32B
🟢	nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base	53.74	71.43	66.12	64.47	64.03	21.51	34.88	12.3	nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base
⭕	mistralai/Ministral-8B-Instruct-2410	53.29	56.43	69.76	63.1	67.99	25.25	37.21	8.02	mistralai/Ministral-8B-Instruct-2410
🟢	Qwen/Qwen3-4B-Base	53.1	72.86	67.82	60.17	61.51	24.71	31.56	4.02	Qwen/Qwen3-4B-Base
🟦	MohamedRashad/Arabic-Orpo-Llama-3-8B-Instruct	52.94	77.86	68.38	52.67	67.63	25.54	25.58	8.03	MohamedRashad/Arabic-Orpo-Llama-3-8B-Instruct
🟢	meta-llama/Meta-Llama-3.1-8B	52.29	64.05	65.41	56.03	66.19	27.52	34.55	8.03	meta-llama/Meta-Llama-3.1-8B
🟦	Danielbrdz/Barcenas-Llama3-8b-ORPO	52.19	74.52	64.28	51.4	64.75	24.61	33.55	8.03	Danielbrdz/Barcenas-Llama3-8b-ORPO
🟢	yam-peleg/Hebrew-Mistral-7B	51.77	59.52	63.61	48.57	64.75	28.98	45.18	7.5	yam-peleg/Hebrew-Mistral-7B
🟢	google/gemma-7b	51.7	59.76	67.43	62.73	62.59	27.93	29.73	8.54	google/gemma-7b
⭕	Qwen/Qwen3-4B	51.33	64.05	67.06	69.07	58.99	21.25	27.57	4.02	Qwen/Qwen3-4B
⭕	yam-peleg/Hebrew-Gemma-11B-Instruct	50.63	55.48	61.33	69.7	59.71	27.32	30.23	10.48	yam-peleg/Hebrew-Gemma-11B-Instruct
🔶	DeepMount00/Llama-3-8b-Ita	50.62	71.67	61.81	51.2	62.95	24.03	32.06	8.03	DeepMount00/Llama-3-8b-Ita
🔶	lightblue/suzume-llama-3-8B-multilingual	50.22	73.1	66.38	50.87	60.79	25.26	24.92	8.03	lightblue/suzume-llama-3-8B-multilingual
🔶	uygarkurt/llama-3-merged-linear	49.88	71.9	63.07	51	63.31	24.31	25.69	8.03	uygarkurt/llama-3-merged-linear
🟦	abacusai/Llama-3-Smaug-8B	49.87	75.95	59.89	46.73	62.23	24.2	30.23	8.03	abacusai/Llama-3-Smaug-8B
🟢	Qwen/Qwen2-7B	49.77	66.67	63.15	54.7	60.07	24.81	29.24	7.62	Qwen/Qwen2-7B

Hebrew LLM Leaderboard