LLVM 23.0.0git
llvm::vputils Namespace Reference

Functions

bool onlyFirstLaneUsed (const VPValue *Def)
 Returns true if only the first lane of Def is used.
bool onlyFirstPartUsed (const VPValue *Def)
 Returns true if only the first part of Def is used.
bool onlyScalarValuesUsed (const VPValue *Def)
 Returns true if only scalar values of Def are used by all users.
VPValuegetOrCreateVPValueForSCEVExpr (VPlan &Plan, const SCEV *Expr)
 Get or create a VPValue that corresponds to the expansion of Expr.
const SCEVgetSCEVExprForVPValue (const VPValue *V, PredicatedScalarEvolution &PSE, const Loop *L=nullptr)
 Return the SCEV expression for V.
bool isAddressSCEVForCost (const SCEV *Addr, ScalarEvolution &SE, const Loop *L)
 Returns true if Addr is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e.
bool isSingleScalar (const VPValue *VPV)
 Returns true if VPV is a single scalar, either because it produces the same value for all lanes or only has its first lane used.
bool isHeaderMask (const VPValue *V, const VPlan &Plan)
 Return true if V is a header mask in Plan.
bool isUniformAcrossVFsAndUFs (const VPValue *V)
 Checks if V is uniform across all VF lanes and UF parts.
VPBasicBlockgetFirstLoopHeader (VPlan &Plan, VPDominatorTree &VPDT)
 Returns the header block of the first, top-level loop, or null if none exist.
unsigned getVFScaleFactor (VPRecipeBase *R)
 Get the VF scaling factor applied to the recipe's output, if the recipe has one.
bool cannotHoistOrSinkRecipe (const VPRecipeBase &R, bool Sinking=false)
 Return true if we do not know how to (mechanically) hoist or sink R.
template<typename Ty>
Intrinsic::ID getIntrinsicID (const Ty *R)
 Return the intrinsic ID underlying a call.
LLVM_ABI_FOR_TEST std::optional< VPValue * > getRecipesForUncountableExit (SmallVectorImpl< VPInstruction * > &Recipes, SmallVectorImpl< VPInstruction * > &GEPs, VPBasicBlock *LatchVPBB)
 Returns the VPValue representing the uncountable exit comparison used by AnyOf if the recipes it depends on can be traced back to live-ins and the addresses (in GEP/PtrAdd form) of any (non-masked) load used in generating the values for the comparison.
std::optional< MemoryLocationgetMemoryLocation (const VPRecipeBase &R)
 Return a MemoryLocation for R with noalias metadata populated from R, if the recipe is supported and std::nullopt otherwise.
VPIRFlags getFlagsFromIndDesc (const InductionDescriptor &ID)
 Extracts and returns NoWrap and FastMath flags from the induction binop in ID.
template<typename PredT>
VPRecipeBasefindRecipe (VPValue *Start, PredT Pred)
 Search Start's users for a recipe satisfying Pred, looking through recipes with definitions.
VPInstructionfindCanonicalIVIncrement (VPlan &Plan)
 Find the canonical IV increment of Plan's vector loop region.
GEPNoWrapFlags getGEPFlagsForPtr (VPValue *Ptr)
 Returns the GEP nowrap flags for Ptr, looking through pointer casts mirroring Value::stripPointerCasts.
bool isUsedByLoadStoreAddress (const VPValue *V)
 Returns true if V is used as part of the address of another load or store.
VPInstructionfindComputeReductionResult (VPReductionPHIRecipe *PhiR)
 Find the ComputeReductionResult recipe for PhiR, looking through selects inserted for predicated reductions or tail folding.
VPSingleDefRecipefindHeaderMask (VPlan &Plan)
 Collect the header mask with the pattern: (ICMP_ULE, WideCanonicalIV, backedge-taken-count) Note: If alias masking is enabled this will find: (AND, HeaderMask, AliasMask) TODO: Introduce explicit recipe for header-mask instead of searching the header-mask pattern manually.
VPValuefindIncomingAliasMask (const VPlan &Plan)
 Finds the incoming alias-mask within the vector preheader.

Function Documentation

◆ cannotHoistOrSinkRecipe()

bool llvm::vputils::cannotHoistOrSinkRecipe ( const VPRecipeBase & R,
bool Sinking = false )

Return true if we do not know how to (mechanically) hoist or sink R.

When sinking, passing Sinking = true ensures that assumes aren't sunk. Returns true for recipes that access memory.

Definition at line 516 of file VPlanUtils.cpp.

References llvm::dyn_cast(), llvm::PatternMatch::m_Intrinsic(), and llvm::SCEVPatternMatch::match().

Referenced by cannotHoistOrSinkRecipe(), hoistPreviousBeforeFORUsers(), licm(), sinkRecurrenceUsersAfterPrevious(), and sinkScalarOperands().

◆ findCanonicalIVIncrement()

◆ findComputeReductionResult()

VPInstruction * llvm::vputils::findComputeReductionResult ( VPReductionPHIRecipe * PhiR)

Find the ComputeReductionResult recipe for PhiR, looking through selects inserted for predicated reductions or tail folding.

Definition at line 837 of file VPlanUtils.cpp.

References llvm::cast(), llvm::VPlanPatternMatch::findUserOf(), llvm::VPHeaderPHIRecipe::getBackedgeValue(), llvm::PatternMatch::m_Select(), and llvm::VPlanPatternMatch::m_VPValue().

Referenced by llvm::VPlanTransforms::createPartialReductions(), and llvm::VPlanTransforms::handleMaxMinNumReductions().

◆ findHeaderMask()

◆ findIncomingAliasMask()

VPValue * llvm::vputils::findIncomingAliasMask ( const VPlan & Plan)

Finds the incoming alias-mask within the vector preheader.

Definition at line 715 of file VPlanUtils.cpp.

References llvm::cast(), llvm::VPlan::getVectorPreheader(), llvm::VPlanPatternMatch::m_VPInstruction(), and llvm::PatternMatch::match().

Referenced by findHeaderMask(), and llvm::VPlanTransforms::materializeAliasMask().

◆ findRecipe()

template<typename PredT>
VPRecipeBase * llvm::vputils::findRecipe ( VPValue * Start,
PredT Pred )
inline

◆ getFirstLoopHeader()

VPBasicBlock * llvm::vputils::getFirstLoopHeader ( VPlan & Plan,
VPDominatorTree & VPDT )

◆ getFlagsFromIndDesc()

VPIRFlags llvm::vputils::getFlagsFromIndDesc ( const InductionDescriptor & ID)
inline

Extracts and returns NoWrap and FastMath flags from the induction binop in ID.

Definition at line 122 of file VPlanUtils.h.

References assert(), llvm::dyn_cast_if_present(), llvm::InductionDescriptor::IK_FpInduction, and llvm::InductionDescriptor::IK_IntInduction.

Referenced by createWidenInductionRecipe().

◆ getGEPFlagsForPtr()

GEPNoWrapFlags llvm::vputils::getGEPFlagsForPtr ( VPValue * Ptr)

Returns the GEP nowrap flags for Ptr, looking through pointer casts mirroring Value::stripPointerCasts.

Definition at line 151 of file VPlanUtils.cpp.

References llvm::any_of(), llvm::drop_begin(), llvm::dyn_cast(), and llvm::GEPNoWrapFlags::none().

Referenced by llvm::VPRecipeBuilder::tryToWidenMemory().

◆ getIntrinsicID()

template<typename Ty>
Intrinsic::ID llvm::vputils::getIntrinsicID ( const Ty * R)

Return the intrinsic ID underlying a call.

Definition at line 80 of file VPlanUtils.h.

References Call, llvm::cast(), llvm::dyn_cast(), F, llvm::isa(), and llvm::Intrinsic::not_intrinsic.

Referenced by llvm::VPlanPatternMatch::IntrinsicID_match::match().

◆ getMemoryLocation()

std::optional< MemoryLocation > llvm::vputils::getMemoryLocation ( const VPRecipeBase & R)

Return a MemoryLocation for R with noalias metadata populated from R, if the recipe is supported and std::nullopt otherwise.

The pointer of the location is conservatively set to nullptr.

Definition at line 761 of file VPlanUtils.cpp.

References llvm::dyn_cast().

Referenced by cannotHoistOrSinkRecipe(), canSinkStoreWithNoAliasCheck(), and llvm::VPlanTransforms::hoistPredicatedLoads().

◆ getOrCreateVPValueForSCEVExpr()

VPValue * llvm::vputils::getOrCreateVPValueForSCEVExpr ( VPlan & Plan,
const SCEV * Expr )

Get or create a VPValue that corresponds to the expansion of Expr.

If Expr is a SCEVConstant or SCEVUnknown, return a VPValue wrapping the live-in value. Otherwise return a VPExpandSCEVRecipe to expand Expr. If Plan's pre-header already contains a recipe expanding Expr, return it. If not, create a new one.

Definition at line 40 of file VPlanUtils.cpp.

References llvm::dyn_cast(), llvm::VPBasicBlock::end(), llvm::VPlan::getEntry(), llvm::VPBasicBlock::getFirstNonPhi(), llvm::VPlan::getOrAddLiveIn(), llvm::VPBasicBlock::insert(), and llvm::isa().

Referenced by addInitialSkeleton(), llvm::VPlanTransforms::convertToStridedAccesses(), createWidenInductionRecipe(), llvm::VPlanTransforms::materializeAliasMask(), and llvm::VPlanTransforms::replaceSymbolicStrides().

◆ getRecipesForUncountableExit()

std::optional< VPValue * > llvm::vputils::getRecipesForUncountableExit ( SmallVectorImpl< VPInstruction * > & Recipes,
SmallVectorImpl< VPInstruction * > & GEPs,
VPBasicBlock * LatchVPBB )

Returns the VPValue representing the uncountable exit comparison used by AnyOf if the recipes it depends on can be traced back to live-ins and the addresses (in GEP/PtrAdd form) of any (non-masked) load used in generating the values for the comparison.

The recipes are stored in Recipes, and recipes forming an address for a load are also added to GEPs.

Definition at line 532 of file VPlanUtils.cpp.

References llvm::cast(), llvm::SmallVectorTemplateCommon< T, typename >::empty(), llvm::VPValue::getDefiningRecipe(), llvm::VPUser::getNumOperands(), llvm::VPBasicBlock::getTerminator(), llvm::VPlanPatternMatch::m_AnyOf(), llvm::VPlanPatternMatch::m_BranchOnTwoConds(), llvm::PatternMatch::m_ICmp(), llvm::VPlanPatternMatch::m_LiveIn(), llvm::VPlanPatternMatch::m_VPInstruction(), llvm::VPlanPatternMatch::m_VPValue(), llvm::SCEVPatternMatch::match(), llvm::SmallVectorImpl< T >::pop_back_val(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by handleUncountableExitsWithSideEffects().

◆ getSCEVExprForVPValue()

const SCEV * llvm::vputils::getSCEVExprForVPValue ( const VPValue * V,
PredicatedScalarEvolution & PSE,
const Loop * L = nullptr )

Return the SCEV expression for V.

Returns SCEVCouldNotCompute if no SCEV expression could be constructed.

Definition at line 170 of file VPlanUtils.cpp.

References AbstractManglingParser< Derived, Alloc >::Ops, llvm::any_of(), llvm::ArrayRef(), assert(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, llvm::dyn_cast(), llvm::SCEV::FlagAnyWrap, llvm::SCEV::FlagNSW, llvm::ScalarEvolution::getAbsExpr(), llvm::ScalarEvolution::getAddExpr(), llvm::ScalarEvolution::getAddRecExpr(), llvm::ScalarEvolution::getConstant(), llvm::ScalarEvolution::getCouldNotCompute(), llvm::ScalarEvolution::getGEPExpr(), llvm::ScalarEvolution::getMinusOne(), llvm::ScalarEvolution::getMinusSCEV(), llvm::ScalarEvolution::getMulExpr(), llvm::ScalarEvolution::getOne(), llvm::ScalarEvolution::getPowerOfTwo(), llvm::PredicatedScalarEvolution::getPredicatedSCEV(), llvm::Type::getScalarType(), llvm::ScalarEvolution::getSCEV(), getSCEVExprForVPValue(), llvm::PredicatedScalarEvolution::getSE(), llvm::ScalarEvolution::getSignExtendExpr(), llvm::ScalarEvolution::getSMaxExpr(), llvm::ScalarEvolution::getSMinExpr(), llvm::ScalarEvolution::getTruncateExpr(), llvm::ScalarEvolution::getTruncateOrSignExtend(), llvm::SCEV::getType(), llvm::Value::getType(), llvm::ScalarEvolution::getUDivExpr(), llvm::ScalarEvolution::getUMaxExpr(), llvm::ScalarEvolution::getUMinExpr(), llvm::ScalarEvolution::getURemExpr(), llvm::ScalarEvolution::getZero(), llvm::ScalarEvolution::getZeroExtendExpr(), llvm::isa(), llvm::IsaPred, llvm::ScalarEvolution::isSCEVable(), IV, llvm::PatternMatch::m_Add(), llvm::PatternMatch::m_APInt(), llvm::VPlanPatternMatch::m_c_BinaryAnd(), llvm::PatternMatch::m_ConstantInt(), llvm::VPlanPatternMatch::m_GetElementPtr(), llvm::PatternMatch::m_Intrinsic(), llvm::PatternMatch::m_Mul(), llvm::MIPatternMatch::m_Not(), llvm::PatternMatch::m_SExt(), llvm::PatternMatch::m_Shl(), llvm::PatternMatch::m_Sub(), llvm::PatternMatch::m_Trunc(), llvm::PatternMatch::m_UDiv(), llvm::PatternMatch::m_URem(), llvm::VPlanPatternMatch::m_VPValue(), llvm::PatternMatch::m_ZExt(), llvm::SCEVPatternMatch::match(), poisonGuaranteesUB(), and llvm::SmallVectorTemplateBase< T, bool >::push_back().

Referenced by llvm::VPlanTransforms::addMinimumIterationCheck(), areAllLoadsDereferenceable(), collectGroupedReplicateMemOps(), llvm::VPlanTransforms::convertToStridedAccesses(), createWidenInductionRecipe(), getAddressAccessSCEV(), getOptimizableIVOf(), getSCEVExprForVPValue(), handleUncountableExitsWithSideEffects(), isConditionTrueViaVFAndUF(), llvm::VPlanTransforms::optimizeFindIVReductions(), llvm::LoopVectorizationPlanner::selectBestEpiloguePlan(), simplifyBranchConditionForVFAndUF(), llvm::VPlanTransforms::simplifyKnownEVL(), and simplifyLiveInsWithSCEV().

◆ getVFScaleFactor()

unsigned llvm::vputils::getVFScaleFactor ( VPRecipeBase * R)

Get the VF scaling factor applied to the recipe's output, if the recipe has one.

Definition at line 500 of file VPlanUtils.cpp.

References assert(), llvm::cast(), llvm::dyn_cast(), getOpcode(), llvm::isa(), and llvm::VPInstruction::ReductionStartVector.

Referenced by llvm::calculateRegisterUsageForPlan().

◆ isAddressSCEVForCost()

bool llvm::vputils::isAddressSCEVForCost ( const SCEV * Addr,
ScalarEvolution & SE,
const Loop * L )

Returns true if Addr is an address SCEV that can be passed to TTI::getAddressComputationCost, i.e.

the address SCEV is loop invariant, an affine AddRec (i.e. induction ), or an add expression of such operands or a sign-extended AddRec.

Definition at line 366 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::dyn_cast(), llvm::ScalarEvolution::isLoopInvariant(), llvm::SCEVPatternMatch::m_SCEV(), llvm::SCEVPatternMatch::m_scev_AffineAddRec(), and llvm::SCEVPatternMatch::match().

Referenced by getAddressAccessSCEV(), and getAddressAccessSCEV().

◆ isHeaderMask()

◆ isSingleScalar()

◆ isUniformAcrossVFsAndUFs()

bool llvm::vputils::isUniformAcrossVFsAndUFs ( const VPValue * V)

Checks if V is uniform across all VF lanes and UF parts.

It is considered as such if it is either loop invariant (defined outside the vector region) or its operands are known to be uniform across all VFs and UFs (e.g. VPDerivedIV or the canonical IV).

Definition at line 441 of file VPlanUtils.cpp.

References llvm::all_of(), llvm::TypeSwitch< T, ResultT >::Case(), llvm::Default, llvm::VPlan::getEntry(), llvm::VPInstruction::getOpcode(), llvm::VPBlockBase::getPlan(), llvm::VPlan::getVectorPreheader(), llvm::isa(), llvm::VPInstruction::isSingleScalar(), isUniformAcrossVFsAndUFs(), llvm::VPInstruction::isVectorToScalar(), llvm::VPlanPatternMatch::m_VPInstruction(), llvm::SCEVPatternMatch::match(), llvm::VPUser::operands(), and preservesUniformity().

Referenced by isKnownMonotonic(), isUniformAcrossVFsAndUFs(), and narrowToSingleScalarRecipes().

◆ isUsedByLoadStoreAddress()

◆ onlyFirstLaneUsed()

◆ onlyFirstPartUsed()

bool llvm::vputils::onlyFirstPartUsed ( const VPValue * Def)

Returns true if only the first part of Def is used.

Definition at line 30 of file VPlanUtils.cpp.

References llvm::all_of().

Referenced by llvm::VPInstruction::usesFirstPartOnly().

◆ onlyScalarValuesUsed()