Namespaces
namespace	detail

Functions
bool	onlyFirstLaneUsed (const VPValue *Def)
	Returns true if only the first lane of `Def` is used.
bool	onlyFirstPartUsed (const VPValue *Def)
	Returns true if only the first part of `Def` is used.
bool	onlyScalarValuesUsed (const VPValue *Def)
	Returns true if only scalar values of `Def` are used by all users.
VPValue *	getOrCreateVPValueForSCEVExpr (VPlan &Plan, const SCEV *Expr)
	Get or create a VPValue that corresponds to the expansion of `Expr`.
const SCEV *	getSCEVExprForVPValue (const VPValue V, PredicatedScalarEvolution &PSE, const Loop L=nullptr)
	Return the SCEV expression for `V`.
bool	isAddressSCEVForCost (const SCEV Addr, ScalarEvolution &SE, const Loop L)
	Returns true if `Addr` is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e.
bool	isSingleScalar (const VPValue *VPV)
	Returns true if `VPV` is a single scalar, either because it produces the same value for all lanes or only has its first lane used.
bool	isUniformAcrossVFsAndUFs (const VPValue *V)
	Checks if `V` is uniform across all VF lanes and UF parts.
bool	isElementwise (const VPValue *V)
	Return true if `V` is elementwise, i.e. none of the lanes are permuted.
bool	doesGeneratePerAllLanes (const VPRecipeBase *R)
	Returns true if `R` produces scalar values for all VF lanes.
VPBasicBlock *	getFirstLoopHeader (VPlan &Plan, VPDominatorTree &VPDT)
	Returns the header block of the first, top-level loop, or null if none exist.
unsigned	getVFScaleFactor (VPRecipeBase *R)
	Get the VF scaling factor applied to the recipe's output, if the recipe has one.
bool	cannotHoistOrSinkRecipe (const VPRecipeBase &R, bool Sinking=false)
	Return true if we do not know how to (mechanically) hoist or sink `R`.
template<typename Ty>
Intrinsic::ID	getIntrinsicID (const Ty *R)
	Return the intrinsic ID underlying a call.
unsigned	getOpcode (const VPValue *V)
	Return the instruction opcode for the recipe defining `V` or 0 for unsupported recipes and VPValues not defined by a recipe.
std::optional< std::pair< bool, unsigned > >	getOpcodeOrIntrinsicID (const VPValue *V)
	Get the instruction opcode or intrinsic ID for the recipe defining `V`.
std::optional< MemoryLocation >	getMemoryLocation (const VPRecipeBase &R)
	Return a MemoryLocation for `R` with noalias metadata populated from `R`, if the recipe is supported and std::nullopt otherwise.
VPIRFlags	getFlagsFromIndDesc (const InductionDescriptor &ID)
	Extracts and returns NoWrap and FastMath flags from the induction binop in `ID`.
template<typename PredT>
VPRecipeBase *	findRecipe (VPValue *Start, PredT Pred)
	Search `Start's` users for a recipe satisfying `Pred`, looking through recipes with definitions.
VPInstruction *	findCanonicalIVIncrement (VPlan &Plan)
	Find the canonical IV increment of `Plan's` vector loop region.
GEPNoWrapFlags	getGEPFlagsForPtr (VPValue *Ptr)
	Returns the GEP nowrap flags for `Ptr`, looking through pointer casts mirroring Value::stripPointerCasts.
bool	isUsedByLoadStoreAddress (const VPValue *V)
	Returns true if `V` is used as part of the address of another load or store.
VPInstruction *	findComputeReductionResult (VPReductionPHIRecipe *PhiR)
	Find the ComputeReductionResult recipe for `PhiR`, looking through selects inserted for predicated reductions or tail folding.
VPValue *	findIncomingAliasMask (const VPlan &Plan)
	Finds the incoming alias-mask within the vector preheader.
SmallVector< std::pair< VPBasicBlock , VPIRBasicBlock > >	getEarlyExits (const VPlan &Plan, const VPBlockBase *MiddleVPBB)
	Returns the (early exiting block, exit block) pairs of `Plan`, i.e.
VPScalarIVStepsRecipe *	createScalarIVSteps (VPlan &Plan, InductionDescriptor::InductionKind Kind, Instruction::BinaryOps InductionOpcode, FPMathOperator FPBinOp, Instruction TruncI, VPValue StartV, VPValue Step, DebugLoc DL, VPBuilder &Builder, const VPIRFlags::WrapFlagsTy &Flags={})
	Create a scalar-iv-steps recipe over `Plan's` canonical IV for an induction of `Kind` with `InductionOpcode` / `FPBinOp`, start value `StartV` and step `Step`, truncated to `TruncI's` type if `TruncI` is non-null, inserting recipes via `Builder`.
VPValue *	scalarizeVPWidenPointerInduction (VPWidenPointerInductionRecipe *PtrIV, VPlan &Plan, VPBuilder &Builder)
	Scalarize a VPWidenPointerInductionRecipe by replacing it with a PtrAdd (IndStart, ScalarIVSteps (0, Step)).
bool	isDeadRecipe (VPRecipeBase &R)
	Returns true if `R` is dead, i.e.
void	recursivelyDeleteDeadRecipes (VPValue *V)
	Recursively delete `V` and any of its operands that become dead.
SmallVector< VPUser * >	collectUsersRecursively (VPValue *V)
	Collect all users of `V`, looking through recipes that define other values.
VPIRValue *	tryToFoldLiveIns (VPSingleDefRecipe &R, ArrayRef< VPValue * > Operands, const DataLayout &DL)
	Try to fold `R` using InstSimplifyFolder.
BranchProbability	getExecutionProbability (BlockFrequency Freq)
	Returns `Freq` as a BranchProbability, relative to AlwaysExecutesFreq.
DenseMap< const VPBasicBlock *, std::optional< BlockFrequency > >	computeExecutionFrequencies (ArrayRef< VPBasicBlock * > Blocks)
	Computes for each block in `Blocks`, which must be in reverse post-order, the frequency with which it executes relative to the first (header) block.
template<typename Match_t, typename Builder>
void	pullOutPermutations (VPlan &Plan, Match_t Perm, Builder Build)
	Removes the permutation pattern `Perm` from any elementwise operations in the plan, by constructing a new permutation via `Build`.

Variables
constexpr uint64_t	AlwaysExecutesFreq = 1ULL << 63
	Denominator of the frequencies computed by computeExecutionFrequencies, i.e.

Function Documentation

◆ cannotHoistOrSinkRecipe()

bool llvm::vputils::cannotHoistOrSinkRecipe	(	const VPRecipeBase &	R,
		bool	Sinking = false )

Return true if we do not know how to (mechanically) hoist or sink R.

When sinking, passing Sinking = true ensures that assumes aren't sunk. Returns true for recipes that access memory.

Definition at line 560 of file VPlanUtils.cpp.

References llvm::dyn_cast(), llvm::PatternMatch::m_Intrinsic(), and llvm::SCEVPatternMatch::match().

Referenced by cannotHoistOrSinkRecipe(), hoistPreviousBeforeFORUsers(), sinkRecurrenceUsersAfterPrevious(), and sinkScalarOperands().

◆ collectUsersRecursively()

SmallVector< VPUser * > llvm::vputils::collectUsersRecursively ( VPValue * V )

Collect all users of V, looking through recipes that define other values.

Definition at line 1152 of file VPlanUtils.cpp.

References llvm::cast(), llvm::VPDef::definedValues(), llvm::from_range, I, and Users.

Referenced by llvm::VPlanTransforms::clearReductionWrapFlags(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::optimizeEVLMasks(), and tryToRemoveDeadCycle().

◆ computeExecutionFrequencies()

DenseMap< const VPBasicBlock *, std::optional< BlockFrequency > > llvm::vputils::computeExecutionFrequencies ( ArrayRef< VPBasicBlock * > Blocks )

Computes for each block in Blocks, which must be in reverse post-order, the frequency with which it executes relative to the first (header) block.

The frequency of a block is the sum over its incoming edges, or std::nullopt if any edge on a path reaching it lacks branch weights.

Definition at line 1227 of file VPlanUtils.cpp.

References AlwaysExecutesFreq, assert(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::at(), llvm::ArrayRef< T >::drop_front(), llvm::ArrayRef< T >::empty(), llvm::ArrayRef< T >::front(), getSuccessorProbabilities(), llvm::DenseMapBase< DerivedT, KeyT, ValueT, KeyInfoT, BucketT >::reserve(), scaleKeepingNonZero(), and llvm::ArrayRef< T >::size().

Referenced by llvm::VPlanTransforms::introduceMasksAndLinearize(), and verifyExecutionFrequenciesMatchBFI().

◆ createScalarIVSteps()

VPScalarIVStepsRecipe * llvm::vputils::createScalarIVSteps	(	VPlan &	Plan,
		InductionDescriptor::InductionKind	Kind,
		Instruction::BinaryOps	InductionOpcode,
		FPMathOperator *	FPBinOp,
		Instruction *	TruncI,
		VPValue *	StartV,
		VPValue *	Step,
		DebugLoc	DL,
		VPBuilder &	Builder,
		const VPIRFlags::WrapFlagsTy &	Flags = {} )

Create a scalar-iv-steps recipe over Plan's canonical IV for an induction of Kind with InductionOpcode / FPBinOp, start value StartV and step Step, truncated to TruncI's type if TruncI is non-null, inserting recipes via Builder.

Definition at line 611 of file VPlanUtils.cpp.

References assert(), llvm::cast(), DL, llvm::VPRegionBlock::getCanonicalIV(), llvm::VPBlockBase::getEntryBasicBlock(), llvm::Type::getScalarSizeInBits(), llvm::VPRecipeValue::getScalarType(), llvm::VPValue::getScalarType(), llvm::VPBlockBase::getSingleHierarchicalPredecessor(), llvm::Value::getType(), llvm::VPlan::getVectorLoopRegion(), llvm::VPlan::getVF(), and llvm::Type::isIntegerTy().

Referenced by legalizeAndOptimizeInductions(), llvm::VPlanTransforms::replaceWideCanonicalIVWithWideIV(), and scalarizeVPWidenPointerInduction().

◆ doesGeneratePerAllLanes()

bool llvm::vputils::doesGeneratePerAllLanes ( const VPRecipeBase * R )

Returns true if R produces scalar values for all VF lanes.

Definition at line 526 of file VPlanUtils.cpp.

References llvm::dyn_cast().

Referenced by llvm::VPlanTransforms::materializePacksAndUnpacks(), llvm::VPlanTransforms::replicateByVF(), sinkScalarOperands(), and llvm::VPRecipeBuilder::tryToCreateWidenNonPhiRecipe().

◆ findCanonicalIVIncrement()

VPInstruction * llvm::vputils::findCanonicalIVIncrement ( VPlan & Plan )

Find the canonical IV increment of Plan's vector loop region.

Returns nullptr if not found.

Definition at line 719 of file VPlanUtils.cpp.

Referenced by llvm::VPRegionBlock::getOrCreateCanonicalIVIncrement(), and preparePlanForEpilogueVectorLoop().

◆ findComputeReductionResult()

VPInstruction * llvm::vputils::findComputeReductionResult ( VPReductionPHIRecipe * PhiR )

Find the ComputeReductionResult recipe for PhiR, looking through selects inserted for predicated reductions or tail folding.

Definition at line 779 of file VPlanUtils.cpp.

References llvm::cast(), llvm::VPlanPatternMatch::findUserOf(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::PatternMatch::m_Select(), and llvm::VPlanPatternMatch::m_VPValue().

Referenced by llvm::VPlanTransforms::createPartialReductions(), and llvm::VPlanTransforms::handleMaxMinNumReductions().

◆ findIncomingAliasMask()

VPValue * llvm::vputils::findIncomingAliasMask ( const VPlan & Plan )

Finds the incoming alias-mask within the vector preheader.

Definition at line 594 of file VPlanUtils.cpp.

References llvm::cast(), llvm::VPlan::getVectorPreheader(), llvm::VPlanPatternMatch::m_VPInstruction(), and llvm::PatternMatch::match().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), llvm::VPlanTransforms::materializeAliasMask(), and llvm::LoopVectorizationPlanner::selectBestEpiloguePlan().

◆ findRecipe()

template<typename PredT>

VPRecipeBase * llvm::vputils::findRecipe	(	VPValue *	Start,
		PredT	Pred )

inline

Search Start's users for a recipe satisfying Pred, looking through recipes with definitions.

Definition at line 151 of file VPlanUtils.h.

References llvm::cast(), llvm::VPValue::getDefiningRecipe(), I, llvm::SetVector< T, Vector, Set, N >::insert(), llvm::SetVector< T, Vector, Set, N >::size(), and llvm::VPValue::users().

Referenced by findFindIVSelect(), llvm::VPlanTransforms::optimizeFindIVReductions(), and preparePlanForEpilogueVectorLoop().

◆ getEarlyExits()

SmallVector< std::pair< VPBasicBlock , VPIRBasicBlock > > llvm::vputils::getEarlyExits	(	const VPlan &	Plan,
		const VPBlockBase *	MiddleVPBB )

Returns the (early exiting block, exit block) pairs of Plan, i.e.

all edges to an exit block that do not come from MiddleVPBB.

Definition at line 602 of file VPlanUtils.cpp.

References llvm::cast(), llvm::SmallVectorImpl< T >::emplace_back(), llvm::VPlan::getExitBlocks(), and llvm::VPBlockBase::getPredecessors().

Referenced by llvm::VPlanTransforms::handleCountableEarlyExits(), and llvm::VPlanTransforms::handleUncountableEarlyExits().

◆ getExecutionProbability()

BranchProbability llvm::vputils::getExecutionProbability ( BlockFrequency Freq )

Returns Freq as a BranchProbability, relative to AlwaysExecutesFreq.

Definition at line 1174 of file VPlanUtils.cpp.

References AlwaysExecutesFreq, getBranchProbabilityKeepingPartial(), and llvm::BlockFrequency::getFrequency().

Referenced by convertFrequencyToBranchWeights(), and verifyExecutionFrequenciesMatchBFI().

◆ getFirstLoopHeader()

VPBasicBlock * llvm::vputils::getFirstLoopHeader	(	VPlan &	Plan,
		VPDominatorTree &	VPDT )

Returns the header block of the first, top-level loop, or null if none exist.

Definition at line 536 of file VPlanUtils.cpp.

References llvm::cast(), llvm::find_if(), llvm::VPlan::getEntry(), I, llvm::VPBlockUtils::isHeader(), and llvm::vp_depth_first_shallow().

Referenced by llvm::LoopVectorizationPlanner::executePlan(), and llvm::InnerLoopVectorizer::fixVectorizedLoop().

◆ getFlagsFromIndDesc()

VPIRFlags llvm::vputils::getFlagsFromIndDesc ( const InductionDescriptor & ID )

inline

Extracts and returns NoWrap and FastMath flags from the induction binop in ID.

Definition at line 134 of file VPlanUtils.h.

References assert(), llvm::dyn_cast_if_present(), llvm::InductionDescriptor::IK_FpInduction, and llvm::InductionDescriptor::IK_IntInduction.

Referenced by createWidenInductionRecipe().

◆ getGEPFlagsForPtr()

GEPNoWrapFlags llvm::vputils::getGEPFlagsForPtr ( VPValue * Ptr )

Returns the GEP nowrap flags for Ptr, looking through pointer casts mirroring Value::stripPointerCasts.

Definition at line 114 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::drop_begin(), llvm::dyn_cast(), llvm::MIPatternMatch::m_ZeroInt(), llvm::PatternMatch::match_fn(), and llvm::GEPNoWrapFlags::none().

Referenced by llvm::VPBuilder::createConsecutiveVectorPointer().

◆ getIntrinsicID()

template<typename Ty>

Intrinsic::ID llvm::vputils::getIntrinsicID ( const Ty * R )

Return the intrinsic ID underlying a call.

Definition at line 87 of file VPlanUtils.h.

References Call, llvm::cast(), llvm::dyn_cast(), F, llvm::VPInstruction::Intrinsic, llvm::isa(), and llvm::Intrinsic::not_intrinsic.

Referenced by llvm::VPInstructionWithType::computeCost(), llvm::VPInstructionWithType::execute(), getOpcodeOrIntrinsicID(), llvm::VPlanPatternMatch::IntrinsicID_match::match(), llvm::VPInstruction::opcodeMayReadOrWriteFromMemory(), and llvm::VPInstructionWithType::printRecipe().

◆ getMemoryLocation()

std::optional< MemoryLocation > llvm::vputils::getMemoryLocation ( const VPRecipeBase & R )

Return a MemoryLocation for R with noalias metadata populated from R, if the recipe is supported and std::nullopt otherwise.

The pointer of the location is conservatively set to nullptr.

Definition at line 706 of file VPlanUtils.cpp.

References llvm::dyn_cast().

Referenced by cannotHoistOrSinkRecipe(), canSinkStoreWithNoAliasCheck(), and llvm::VPlanTransforms::hoistPredicatedLoads().

◆ getOpcode()

unsigned llvm::vputils::getOpcode ( const VPValue * V )

Return the instruction opcode for the recipe defining V or 0 for unsupported recipes and VPValues not defined by a recipe.

Definition at line 381 of file VPlanUtils.cpp.

References llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, I, and llvm::VPInstruction::OpsEnd.

Referenced by canNarrowOps(), getOpcodeOrIntrinsicID(), getVFScaleFactor(), isElementwise(), narrowToSingleScalarRecipes(), and poisonGuaranteesUB().

◆ getOpcodeOrIntrinsicID()

std::optional< std::pair< bool, unsigned > > llvm::vputils::getOpcodeOrIntrinsicID ( const VPValue * V )

Get the instruction opcode or intrinsic ID for the recipe defining V.

Returns an optional pair, where the first element indicates whether it is an intrinsic ID.

Definition at line 397 of file VPlanUtils.cpp.

References getIntrinsicID(), and getOpcode().

Referenced by tryToFoldLiveIns().

◆ getOrCreateVPValueForSCEVExpr()

VPValue * llvm::vputils::getOrCreateVPValueForSCEVExpr	(	VPlan &	Plan,
		const SCEV *	Expr )

Get or create a VPValue that corresponds to the expansion of Expr.

If Expr is a SCEVConstant or SCEVUnknown, return a VPValue wrapping the live-in value. Otherwise return a VPExpandSCEVRecipe to expand Expr. If Plan's pre-header already contains a recipe expanding Expr, return it. If not, create a new one.

Definition at line 47 of file VPlanUtils.cpp.

References llvm::dyn_cast(), llvm::VPBasicBlock::end(), llvm::VPlan::getEntry(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::VPlan::getOrAddLiveIn(), llvm::VPBasicBlock::insert(), and llvm::isa().

Referenced by addInitialSkeleton(), createWidenInductionRecipe(), llvm::VPSCEVExpander::expand(), llvm::VPlanTransforms::materializeAliasMask(), and llvm::VPlanTransforms::replaceSymbolicStrides().

◆ getSCEVExprForVPValue()

const SCEV * llvm::vputils::getSCEVExprForVPValue	(	const VPValue *	V,
		PredicatedScalarEvolution &	PSE,
		const Loop *	L = nullptr )

Return the SCEV expression for V.

Returns SCEVCouldNotCompute if no SCEV expression could be constructed.

Definition at line 132 of file VPlanUtils.cpp.

Referenced by llvm::VPlanTransforms::addMinimumIterationCheck(), llvm::VPlanTransforms::areAllLoadsDereferenceable(), collectGroupedReplicateMemOps(), llvm::VPlanTransforms::convertToStridedAccesses(), createWidenInductionRecipe(), getAddressAccessSCEV(), getConstantStride(), getOptimizableIVOf(), getSCEVExprForVPValue(), handleUncountableExitsWithSideEffects(), isConditionTrueViaVFAndUF(), llvm::VPlanTransforms::makeMemOpWideningDecisions(), llvm::VPlanTransforms::optimizeFindIVReductions(), optimizeLatchExitIVUserViaSCEV(), llvm::LoopVectorizationPlanner::selectBestEpiloguePlan(), simplifyBranchConditionForVFAndUF(), llvm::VPlanTransforms::simplifyKnownEVL(), and llvm::VPlanTransforms::simplifyLiveInsWithSCEV().

◆ getVFScaleFactor()

unsigned llvm::vputils::getVFScaleFactor ( VPRecipeBase * R )

Get the VF scaling factor applied to the recipe's output, if the recipe has one.

Definition at line 544 of file VPlanUtils.cpp.

References assert(), llvm::cast(), llvm::dyn_cast(), getOpcode(), llvm::isa(), and llvm::VPInstruction::ReductionStartVector.

Referenced by llvm::calculateRegisterUsageForPlan().

◆ isAddressSCEVForCost()

bool llvm::vputils::isAddressSCEVForCost	(	const SCEV *	Addr,
		ScalarEvolution &	SE,
		const Loop *	L )

Returns true if Addr is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e.

the address SCEV is loop invariant, an affine AddRec (i.e. induction ), or an add expression of such operands or a sign-extended AddRec.

Definition at line 364 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::dyn_cast(), llvm::ScalarEvolution::isLoopInvariant(), llvm::SCEVPatternMatch::m_SCEV(), llvm::SCEVPatternMatch::m_scev_AffineAddRec(), and llvm::SCEVPatternMatch::match().

Referenced by getAddressAccessSCEV(), and getAddressAccessSCEV().

◆ isDeadRecipe()

bool llvm::vputils::isDeadRecipe ( VPRecipeBase & R )

Returns true if R is dead, i.e.

none of its defined values are used and it has no side effects (with the exception of conditional assumes, which are considered dead as their conditions may be flattened).

Definition at line 1112 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::dyn_cast(), llvm::isa(), llvm::PatternMatch::m_Intrinsic(), and llvm::PatternMatch::match().

Referenced by narrowToSingleScalarRecipes(), recursivelyDeleteDeadRecipes(), llvm::VPlanTransforms::removeDeadRecipes(), and llvm::VPlanTransforms::simplifyRecipes().

◆ isElementwise()

bool llvm::vputils::isElementwise ( const VPValue * V )

Return true if V is elementwise, i.e. none of the lanes are permuted.

Definition at line 426 of file VPlanUtils.cpp.

References getOpcode(), llvm::isa(), llvm::Instruction::isBinaryOp(), and llvm::Instruction::isUnaryOp().

Referenced by llvm::vputils::detail::pullOutPermutationsImpl().

◆ isSingleScalar()

bool llvm::vputils::isSingleScalar ( const VPValue * VPV )

Returns true if VPV is a single scalar, either because it produces the same value for all lanes or only has its first lane used.

Definition at line 434 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::dyn_cast(), llvm::VPValue::getDefiningRecipe(), llvm::isa(), llvm::VPRegionBlock::isReplicator(), isSingleScalar(), llvm::VPUser::operands(), and preservesUniformity().

Referenced by cloneForLane(), llvm::VPWidenMemoryRecipe::computeCost(), convertRecipesInRegionBlocksToSingleScalar(), llvm::VPIRPhi::execute(), llvm::VPWidenGEPRecipe::execute(), llvm::VPWidenRecipe::execute(), llvm::VPTransformState::get(), isSingleScalar(), legalizeAndOptimizeInductions(), llvm::VPlanTransforms::materializePacksAndUnpacks(), narrowToSingleScalarRecipes(), simplifyRecipe(), sinkScalarOperands(), llvm::VPRecipeBuilder::tryToCreateWidenNonPhiRecipe(), and llvm::VPWidenGEPRecipe::usesFirstLaneOnly().

◆ isUniformAcrossVFsAndUFs()

bool llvm::vputils::isUniformAcrossVFsAndUFs ( const VPValue * V )

Checks if V is uniform across all VF lanes and UF parts.

It is considered as such if it is either loop invariant (defined outside the vector region) or its operands are known to be uniform across all VFs and UFs (e.g. VPDerivedIV or the canonical IV).

Definition at line 473 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, llvm::dyn_cast(), llvm::VPlan::getEntry(), llvm::VPInstruction::getOpcode(), llvm::VPBlockBase::getPlan(), llvm::VPlan::getVectorPreheader(), llvm::isa(), llvm::VPInstruction::isSingleScalar(), isUniformAcrossVFsAndUFs(), llvm::VPInstruction::isVectorToScalar(), llvm::VPlanPatternMatch::m_ExtractVectorForPart(), llvm::VPlanPatternMatch::m_VPInstruction(), llvm::VPlanPatternMatch::m_VPValue(), llvm::SCEVPatternMatch::match(), llvm::VPUser::operands(), and preservesUniformity().