TFTF (Trusted Firmware Test Framework)

tftf
Data as of 29 Oct 2026, 18:19 UTCLast synced:LAVA 29 Oct 2026, 18:19 UTCJenkins 29 Oct 2026, 18:22 UTCS3 export 7 Oct 2026, 09:53 UTC

Cases

243 cases. 4 failures in the window (last 30 days) across 113,519 assertion results.

Failing and flaky cases first. Pass rate counts passed and failed attempt results, including retries. Skipped and unknown results are excluded. Verdicts summarise the last 30 days.
Case
Role
Pass rate
Failures
Skipped
Unknown
Verdict
tftf/query_vendor_specific_service
assertion
98.2% of 219
4
4
0
stable
tftf/access_from_a_sp_to_a_realm_region
assertion
no verdict
0
47
0
no verdict
tftf/access_mpam_registers
assertion
no verdict
0
221
0
no verdict
tftf/access_pointer_authentication_registers
assertion
no verdict
0
387
0
no verdict
tftf/access_ras_registers
assertion
no verdict
0
191
0
no verdict
tftf/affinity_info_level0_off
assertion
100.0% of 2,680
0
0
0
stable
tftf/affinity_info_level0_on
assertion
100.0% of 2,712
0
1
0
stable
tftf/affinity_info_level0_powerdown
assertion
100.0% of 2,825
0
0
0
stable
tftf/affinity_info_level1_off
assertion
100.0% of 2,601
0
8
0
stable
tftf/affinity_info_level1_on
assertion
100.0% of 2,626
0
1
0
stable
tftf/affinity_info_level2
assertion
100.0% of 2,570
0
16
0
stable
tftf/affinity_info_level3
assertion
100.0% of 2,211
0
360
0
stable
tftf/afp_support
assertion
no verdict
0
384
0
no verdict
tftf/aie_support
assertion
no verdict
0
146
0
no verdict
tftf/amuv1_group_1_raz
assertion
no verdict
0
215
0
no verdict
tftf/amuv1_suspend_resume
assertion
no verdict
0
381
0
no verdict
tftf/amuv1_valid_counter_values
assertion
no verdict
0
387
0
no verdict
tftf/amuv1_world_switch
assertion
no verdict
0
144
0
no verdict
tftf/bigger_ffa_version_than_spm
assertion
no verdict
0
45
0
no verdict
tftf/check_dma_command_by_smmuv3testengine_completes
assertion
no verdict
0
47
0
no verdict
tftf/check_for_mte_register_leakage
assertion
no verdict
0
385
0
no verdict
tftf/check_for_pointer_authentication_key_leakage_from_el3
assertion
no verdict
0
389
0
no verdict
tftf/check_for_pointer_authentication_key_leakage_from_tsp
assertion
no verdict
0
388
0
no verdict
tftf/check_that_simd_registers_context_is_preserved
assertion
no verdict
0
47
0
no verdict
tftf/check_that_sve_operations_in_nwd_are_unaffected_by_swd
assertion
no verdict
0
47
0
no verdict
tftf/check_that_sve_registers_context_is_preserved
assertion
no verdict
0
47
0
no verdict
tftf/cleanup_failure_fails_testcase
assertion
100.0% of 7
0
0
0
stable
tftf/cntfrq_compare_test
assertion
100.0% of 2,837
0
0
0
stable
tftf/context_id_passing
assertion
100.0% of 2,630
0
0
0
stable
tftf/cpu_already_on
assertion
100.0% of 2,609
0
0
0
stable
tftf/cpu_hotplug
assertion
100.0% of 2,793
0
0
0
stable
tftf/cpu_off_on_all_non_lead_cores_in_sequence_and_suspend_lead_to_deepest_power_level
assertion
100.0% of 727
0
367
0
stable
tftf/cpu_suspend_on_all_cores_in_parallel
assertion
100.0% of 724
0
366
0
stable
tftf/cpu_suspend_on_all_cores_in_sequence
assertion
100.0% of 722
0
362
0
stable
tftf/cpu_suspend_to_powerdown_at_level_0
assertion
100.0% of 2,838
0
0
0
stable
tftf/cpu_suspend_to_powerdown_at_level_1
assertion
100.0% of 2,838
0
0
0
stable
tftf/cpu_suspend_to_powerdown_at_level_2
assertion
100.0% of 2,834
0
4
0
stable
tftf/cpu_suspend_to_powerdown_at_level_3
assertion
no verdict
0
2834
0
no verdict
tftf/cpu_suspend_to_standby_at_level_0
assertion
100.0% of 2,838
0
0
0
stable
tftf/cpu_suspend_to_standby_at_level_1
assertion
100.0% of 2,838
0
0
0
stable
tftf/cpu_suspend_to_standby_at_level_2
assertion
100.0% of 2,828
0
10
0
stable
tftf/cpu_suspend_to_standby_at_level_3
assertion
no verdict
0
2828
0
no verdict
tftf/create_a_power_state_with_valid_and_invalid_local_state_id_at_different_levels_and_validate_power_state_parsing
assertion
no verdict
0
389
0
no verdict
tftf/create_all_power_states_and_validate_el3_power_state_parsing
assertion
no verdict
0
389
0
no verdict
tftf/create_invalid_local_power_state_at_all_levels_and_validate_el3_power_state_parsing
assertion
no verdict
0
389
0
no verdict
tftf/create_invalid_power_level_and_validate_el3_power_state_parsing_for_original_state_format
assertion
no verdict
0
389
0
no verdict
tftf/create_invalid_power_state_type_and_validate_el3_power_state_parsing
assertion
no verdict
0
388
0
no verdict
tftf/create_only_local_power_state_and_validate_el3_power_state_parsing
assertion
no verdict
0
390
0
no verdict
tftf/d128_support
assertion
no verdict
0
189
0
no verdict
tftf/debugv8p9_support
assertion
no verdict
0
216
0
no verdict
tftf/donate_memory_to_secure_world
assertion
no verdict
0
47
0
no verdict
tftf/em_cpu_features
assertion
no verdict
0
258
0
no verdict
tftf/events_api
assertion
100.0% of 2,834
0
0
0
stable
tftf/expose_filesystem
assertion
no verdict
0
307
0
no verdict
tftf/fast_smc_while_tsp_preempted
assertion
no verdict
0
361
0
no verdict
tftf/fast_smc32
assertion
100.0% of 388
0
0
0
stable
tftf/fast_smc64
assertion
100.0% of 390
0
0
0
stable
tftf/features
assertion
100.0% of 548
0
9
0
stable
tftf/ff_a_direct_messaging
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_request_sp_to_sp_direct_messaging
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_request_sp_to_sp_direct_messaging_deadlock
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_rxtx_map_api_consecutive
assertion
no verdict
0
46
0
no verdict
tftf/ff_a_rxtx_map_api_success
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_rxtx_remap_unmapped_region_success
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_rxtx_unmap_api_consecutive
assertion
no verdict
0
47
0
no verdict
tftf/ff_a_rxtx_unmap_api_success
assertion
no verdict
0
46
0
no verdict
tftf/ff_a_sp_hotplug
assertion
no verdict
0
47
0
no verdict
tftf/for_stats_after_system_reset
assertion
100.0% of 613
0
0
0
stable
tftf/for_valid_composite_state_cpu_suspend
assertion
100.0% of 2,838
0
0
0
stable
tftf/fwu_auth_failure
assertion
100.0% of 300
0
0
0
stable
tftf/fwu_toc_invalid
assertion
100.0% of 300
0
0
0
stable
tftf/hacdbs_support
assertion
no verdict
0
147
0
no verdict
tftf/hdbss_support
assertion
no verdict
0
148
0
no verdict
tftf/hotplug_a_cluster_in_a_large_iteration_to_stress_cluster_on_and_off_functionality
assertion
100.0% of 2,420
0
0
0
stable
tftf/invalid_cpu
assertion
100.0% of 2,491
0
0
0
stable
tftf/invalid_entry_point
assertion
100.0% of 2,580
0
0
0
stable
tftf/irq_handling
assertion
100.0% of 2,838
0
0
0
stable
tftf/leak_pmu_cycle_counter_values_from_el3_on_psci_suspend_smc
assertion
no verdict
0
390
0
no verdict
tftf/leak_pmu_cycle_counter_values_from_s_el1_on_fast_smc_add
assertion
no verdict
0
390
0
no verdict
tftf/leak_pmu_pc_write_retired_counter_values_from_el3_on_psci_suspend_smc
assertion
no verdict
0
390
0
no verdict
tftf/leak_pmu_pc_write_retired_counter_values_from_s_el1_on_fast_smc_add
assertion
no verdict
0
390
0
no verdict
tftf/lend_memory_clear_flag_set
assertion
no verdict
0
47
0
no verdict
tftf/lend_memory_to_secure_world
assertion
no verdict
0
47
0
no verdict
tftf/ls64_accdata_support
assertion
no verdict
0
220
0
no verdict
tftf/ls64_support
assertion
no verdict
0
222
0
no verdict
tftf/mpam_pe_side_bandwidth_control_support
assertion
no verdict
0
190
0
no verdict
tftf/multi_cpu_realm_payload_delegate_and_undelegate
assertion
no verdict
0
47
0
no verdict
tftf/multicore_spurious_interrupt_test
assertion
no verdict
0
385
0
no verdict
tftf/notifications_bind_unbind_of_bound_notifications
assertion
no verdict
0
47
0
no verdict
tftf/notifications_bind_unbind_sps_spoofing_receiver
assertion
no verdict
0
47
0
no verdict
tftf/notifications_bitmap_create_after_create
assertion
no verdict
0
47
0
no verdict
tftf/notifications_bitmap_create_and_destroy
assertion
no verdict
0
47
0
no verdict
tftf/notifications_bitmap_destroy_not_created
assertion
no verdict
0
46
0
no verdict
tftf/notifications_info_get_no_data
assertion
no verdict
0
47
0
no verdict
tftf/notifications_interrupts_id_retrieval_with_ffa_features
assertion
no verdict
0
47
0
no verdict
tftf/notifications_mp_sp_signals_up_sp_per_vcpu
assertion
no verdict
0
46
0
no verdict
tftf/notifications_sp_signals_sp
assertion
no verdict
0
47
0
no verdict
tftf/notifications_sp_signals_sp_per_vcpu
assertion
no verdict
0
47
0
no verdict
tftf/notifications_sp_signals_sp_with_delayed_sri
assertion
no verdict
0
47
0
no verdict
tftf/notifications_sp_signals_sp_with_immediate_sri
assertion
no verdict
0
47
0
no verdict
tftf/notifications_sp_signals_vm
assertion
no verdict
0
46
0
no verdict
tftf/notifications_sp_signals_vm_per_vcpu
assertion
no verdict
0
47
0
no verdict
tftf/notifications_unbind_while_pending
assertion
no verdict
0
47
0
no verdict
tftf/notifications_vm_signals_sp
assertion
no verdict
0
47
0
no verdict
tftf/notifications_vm_signals_sp_per_vcpu
assertion
no verdict
0
47
0
no verdict
tftf/notifications_zeroed_in_bind_and_unbind
assertion
no verdict
0
46
0
no verdict
tftf/nvm_serialisation
assertion
100.0% of 2,807
0
0
0
stable
tftf/nvm_support
assertion
100.0% of 2,837
0
0
0
stable
tftf/out_of_bounds_accesses
assertion
no verdict
0
192
0
no verdict
tftf/pfar_support
assertion
no verdict
0
148
0
no verdict
tftf/pmuv3_cycle_counter_functional_in_ns
assertion
no verdict
0
296
0
no verdict
tftf/pmuv3_event_counter_functional_in_ns
assertion
no verdict
0
293
0
no verdict
tftf/pmuv3_smc_counter_preservation
assertion
no verdict
0
295
0
no verdict
tftf/probe_pmf_version
assertion
100.0% of 219
0
4
0
stable
tftf/psci_cpu_on_off_stress_test
assertion
100.0% of 2,420
0
0
0
stable
tftf/psci_cpu_on_off_suspend_stress_test
assertion
100.0% of 302
0
0
0
stable
tftf/psci_features
assertion
100.0% of 363
0
3
0
stable
tftf/psci_invalid_features
assertion
100.0% of 337
0
3
0
stable
tftf/psci_mem_protect
assertion
100.0% of 613
0
0
0
stable
tftf/psci_mem_protect_check
assertion
100.0% of 351
0
11
0
stable
tftf/psci_migrate_info_type
assertion
100.0% of 358
0
2
0
stable
tftf/psci_reset2_invalid_reset_options
assertion
100.0% of 613
0
0
0
stable
tftf/psci_version
assertion
100.0% of 2,743
0
3
0
stable
tftf/psci_version_call_on_all_cores_in_parallel
assertion
100.0% of 737
0
364
0
stable
tftf/query_standard_service
assertion
100.0% of 1,409
0
0
0
stable
tftf/random_hotplug_cores_in_a_large_iteration_to_stress_boot_path_code
assertion
100.0% of 2,420
0
0
0
stable
tftf/realm_el1_creation_and_execution_test
assertion
no verdict
0
47
0
no verdict
tftf/realm_payload_boot
assertion
no verdict
0
47
0
no verdict
tftf/realm_payload_delegate_and_undelegate
assertion
no verdict
0
47
0
no verdict
tftf/realm_payload_multi_cpu_request
assertion
no verdict
0
47
0
no verdict
tftf/repeated_hotplug_of_all_cores_to_stress_test_cpu_on_and_cpu_off
assertion
100.0% of 2,420
0
0
0
stable
tftf/repeated_shutdown_of_all_cores_to_stress_test_cpu_on_cpu_suspend_and_cpu_off
assertion
100.0% of 2,420
0
0
0
stable
tftf/request_a_valid_state_switch
assertion
no verdict
0
387
0
no verdict
tftf/request_a_valid_state_switch_after_cpu_on
assertion
no verdict
0
387
0
no verdict
tftf/request_donate_memory_sp_to_sp
assertion
no verdict
0
47
0
no verdict
tftf/request_lend_memory_sp_to_sp
assertion
no verdict
0
47
0
no verdict
tftf/request_lend_memory_sp_to_vm
assertion
no verdict
0
47
0
no verdict
tftf/request_share_memory_sp_to_sp
assertion
no verdict
0
47
0
no verdict
tftf/request_share_memory_sp_to_vm
assertion
no verdict
0
47
0
no verdict
tftf/request_share_ns_memory_large_pa_sp_to_sp
assertion
no verdict
0
47
0
no verdict
tftf/request_state_switch_with_invalid_context
assertion
no verdict
0
386
0
no verdict
tftf/request_state_switch_with_invalid_pc
assertion
no verdict
0
387
0
no verdict
tftf/resume_preempted_std_smc
assertion
no verdict
0
383
0
no verdict
tftf/resume_preempted_std_smc_after_psci_cpu_off_on_cycle
assertion
no verdict
0
384
0
no verdict
tftf/resume_preempted_std_smc_from_other_cpus
assertion
no verdict
0
381
0
no verdict
tftf/resume_smc_without_tsp_preemption
assertion
no verdict
0
375
0
no verdict
tftf/resume_std_smc_from_different_cpus
assertion
no verdict
0
381
0
no verdict
tftf/rnd
assertion
100.0% of 519
0
21
0
stable
tftf/same_ffa_version_as_spm
assertion
no verdict
0
47
0
no verdict
tftf/sctlr2_support
assertion
no verdict
0
189
0
no verdict
tftf/sdei_event_handler_pstate_testing
assertion
no verdict
0
350
0
no verdict
tftf/sdei_event_handler_state_machine_testing
assertion
no verdict
0
367
0
no verdict
tftf/sdei_event_handling_on_all_cores_in_parallel
assertion
no verdict
0
367
0
no verdict
tftf/sdei_event_handling_on_all_cores_in_sequence
assertion
no verdict
0
362
0
no verdict
tftf/sdei_event_routing_all_spi_events_routed_to_all_cpus
assertion
no verdict
0
366
0
no verdict
tftf/sdei_event_signaling_each_core_signals_itself
assertion
100.0% of 1
0
363
0
stable
tftf/sdei_event_signaling_one_core_signals_all_others
assertion
no verdict
0
364
0
no verdict
tftf/sdei_event_signaling_state_registered_enabled_unmasked
assertion
no verdict
0
172
0
no verdict
tftf/setting_bit_31_in_input_version
assertion
no verdict
0
47
0
no verdict
tftf/setup_failure_skips_body_and_runs_cleanup
assertion
100.0% of 7
0
0
0
stable
tftf/sgi_support
assertion
100.0% of 2,827
0
0
0
stable
tftf/share_memory_with_secure_world
assertion
no verdict
0
47
0
no verdict
tftf/smaller_ffa_version_than_spm
assertion
no verdict
0
47
0
no verdict
tftf/smccc_arch_feature_availability_test
assertion
no verdict
0
190
0
no verdict
tftf/smccc_arch_soc_id_test
assertion
100.0% of 289
0
96
0
stable
tftf/smccc_arch_workaround_1_latency
assertion
100.0% of 1
0
389
0
stable
tftf/smccc_arch_workaround_1_test
assertion
no verdict
0
386
0
no verdict
tftf/smccc_arch_workaround_2_test
assertion
100.0% of 1
0
385
0
stable
tftf/smccc_arch_workaround_3_test
assertion
no verdict
0
386
0
no verdict
tftf/smccc_arch_workaround_4_test
assertion
100.0% of 2
0
382
0
stable
tftf/sme_support
assertion
no verdict
0
384
0
no verdict
tftf/sme2_support
assertion
no verdict
0
291
0
no verdict
tftf/sp_notifications_bind_and_unbind
assertion
no verdict
0
47
0
no verdict
tftf/spe_exc_support
assertion
no verdict
0
146
0
no verdict
tftf/spe_nvm_support
assertion
no verdict
0
145
0
no verdict
tftf/spe_support
assertion
no verdict
0
381
0
no verdict
tftf/srmask_support
assertion
no verdict
0
147
0
no verdict
tftf/stats_test_cases_for_cpu_off
assertion
100.0% of 2,838
0
0
0
stable
tftf/std_smc_abortion_while_tsp_preempted
assertion
no verdict
0
368
0
no verdict
tftf/std_smc_resumption_while_tsp_preempted
assertion
no verdict
0
374
0
no verdict
tftf/step2_support
assertion
no verdict
0
147
0
no verdict
tftf/stress_test_the_timer_framework
assertion
100.0% of 1,851
0
0
0
stable
tftf/stress_test_tsp_functionality
assertion
no verdict
0
387
0
no verdict
tftf/stress_tsp_preemption_and_resumption
assertion
no verdict
0
375
0
no verdict
tftf/suspend_to_deepest_power_level_on_all_cores_in_parallel
assertion
100.0% of 725
0
377
0
stable
tftf/suspend_to_deepest_power_level_on_all_cores_in_sequence
assertion
100.0% of 723
0
373
0
stable
tftf/sve_support
assertion
no verdict
0
376
0
no verdict
tftf/system_reset
assertion
100.0% of 613
0
0
0
stable
tftf/system_reset_before_state_switch
assertion
no verdict
0
390
0
no verdict
tftf/target_timer_to_a_power_down_cpu
assertion
100.0% of 2,838
0
0
0
stable
tftf/test_access_to_hcrx_el2
assertion
no verdict
0
147
0
no verdict
tftf/test_binding_more_events_than_are_available_in_the_ppi_range
assertion
no verdict
0
210
0
no verdict
tftf/test_cluster_power_up_latency
assertion
100.0% of 389
0
0
0
stable
tftf/test_ffa_features
assertion
no verdict
0
44
0
no verdict
tftf/test_ffa_partition_info_get
assertion
no verdict
0
46
0
no verdict
tftf/test_ffa_partition_info_get_v1_0
assertion
no verdict
0
47
0
no verdict
tftf/test_ffa_spm_id_get
assertion
no verdict
0
47
0
no verdict
tftf/test_if_rndr_instruction_clears_pstate_nzcv
assertion
no verdict
0
149
0
no verdict
tftf/test_if_rndr_instruction_traps_to_el3_and_gets_a_random_number
assertion
no verdict
0
149
0
no verdict
tftf/test_if_rndrrs_instruction_clears_pstate_nzcv
assertion
no verdict
0
149
0
no verdict
tftf/test_if_rndrss_instruction_traps_to_el3_and_gets_a_random_number
assertion
no verdict
0
149
0
no verdict
tftf/test_managed_exit
assertion
no verdict
0
47
0
no verdict
tftf/test_managed_exit_in_sp_call_chain
assertion
no verdict
0
47
0
no verdict
tftf/test_ns_interrupt_signalable
assertion
no verdict
0
47
0
no verdict
tftf/test_pstate_bits_maintained_on_exception
assertion
no verdict
0
387
0
no verdict
tftf/test_scenario_where_multiple_cpus_call_same_timeout
assertion
100.0% of 2,838
0
0
0
stable
tftf/test_secure_fiq_while_tsp_is_preempted
assertion
no verdict
0
379
0
no verdict
tftf/test_secure_interrupt_handling_while_sp_blocked
assertion
no verdict
0
47
0
no verdict
tftf/test_secure_interrupt_handling_while_sp_running
assertion
no verdict
0
47
0
no verdict
tftf/test_secure_interrupt_handling_while_sp_waiting
assertion
no verdict
0
47
0
no verdict
tftf/test_secure_interrupt_handling_while_sp1_waiting_sp2_running
assertion
no verdict
0
47
0
no verdict
tftf/test_spx_with_managed_exit_and_spy_with_ns_interrupt_signaled
assertion
no verdict
0
47
0
no verdict
tftf/test_spx_with_ns_interrupt_signaled_and_spy_with_managed_exit
assertion
no verdict
0
47
0
no verdict
tftf/test_tftf_can_call_rmm_trp_and_spm_parallelly_on_a_multi_core
assertion
no verdict
0
47
0
no verdict
tftf/test_tftf_can_call_rmm_trp_and_spm_serially_on_a_single_core
assertion
no verdict
0
47
0
no verdict
tftf/test_trace_buffer_management_profiling_exceptions
assertion
no verdict
0
149
0
no verdict
tftf/test_wfet_instruction
assertion
no verdict
0
386
0
no verdict
tftf/test_wfit_instruction
assertion
no verdict
0
384
0
no verdict
tftf/testcase_hooks_succeed
assertion
100.0% of 7
0
0
0
stable
tftf/testing_delegation_fails
assertion
no verdict
0
47
0
no verdict
tftf/tests_for_node_hw_state
assertion
100.0% of 389
0
1
0
stable
tftf/tests_for_node_hw_state_on_multicluster
assertion
100.0% of 385
0
3
0
stable
tftf/testsecurepayload_standard_functions_service_call
assertion
no verdict
0
386
0
no verdict
tftf/the_support
assertion
no verdict
0
188
0
no verdict
tftf/tsp_preempt_by_irq_and_resume
assertion
no verdict
0
362
0
no verdict
tftf/unknown_smc
assertion
100.0% of 1,422
0
0
0
stable
tftf/use_branch_record_buffer_control_registers
assertion
no verdict
0
387
0
no verdict
tftf/use_ecv_registers
assertion
no verdict
0
383
0
no verdict
tftf/use_fgt_registers
assertion
no verdict
0
388
0
no verdict
tftf/use_fgt2_registers
assertion
no verdict
0
220
0
no verdict
tftf/use_fpmr_register
assertion
no verdict
0
191
0
no verdict
tftf/use_mte_instructions
assertion
no verdict
0
385
0
no verdict
tftf/use_pointer_authentication_instructions
assertion
no verdict
0
385
0
no verdict
tftf/use_trace_buffer_control_registers
assertion
no verdict
0
384
0
no verdict
tftf/use_trace_filter_control_registers
assertion
no verdict
0
386
0
no verdict
tftf/use_trace_system_registers
assertion
no verdict
0
384
0
no verdict
tftf/verify_psci_cpu_on_race
assertion
100.0% of 302
0
0
0
stable
tftf/verify_the_timer_interrupt_generation
assertion
100.0% of 2,795
0
0
0
stable
tftf/version
assertion
100.0% of 572
0
25
0
stable
tftf/vm_notifications_bind_and_unbind
assertion
no verdict
0
47
0
no verdict
tftf/vm_notifications_bind_ns_sender
assertion
no verdict
0
47
0
no verdict
tftf/yielding_smc32
assertion
100.0% of 385
0
1
0
stable
tftf/yielding_smc64
assertion
100.0% of 384
0
2
0
stable
Verdicts: flaky attempts of the same build disagreed, or nightly results flipped at least twice in at least 5 evaluations; failing the last two final results failed; stable otherwise. Lab failures are not test results and never count as flips.