snorkel.labeling.lf.nlp_spark.SparkNLPLabelingFunction¶
-
class
snorkel.labeling.lf.nlp_spark.
SparkNLPLabelingFunction
(name, f, resources=None, pre=None, text_field='text', doc_field='doc', language='en_core_web_sm', disable=None, memoize=True)[source]¶ Bases:
snorkel.labeling.lf.nlp.BaseNLPLabelingFunction
Special labeling function type for SpaCy-based LFs running on Spark.
This class is a Spark-compatible version of
NLPLabelingFunction
. SeeNLPLabelingFunction
for details.- Parameters
name (
str
) – Name of the LFf (
Callable
[…,int
]) – Function that implements the core LF logicresources (
Optional
[Mapping
[str
,Any
]]) – Labeling resources passed in tof
viakwargs
pre (
Optional
[List
[BaseMapper
]]) – Preprocessors to run before SpacyPreprocessor is executedtext_field (
str
) – Name of data point text field to inputdoc_field (
str
) – Name of data point field to output parsed document tolanguage (
str
) – SpaCy model to load See https://spacy.io/usage/models#usagedisable (
Optional
[List
[str
]]) – List of pipeline components to disable See https://spacy.io/usage/processing-pipelines#disablingmemoize (
bool
) – Memoize preprocessor outputs?
- Raises
ValueError – Calling incorrectly defined preprocessors
-
__init__
(name, f, resources=None, pre=None, text_field='text', doc_field='doc', language='en_core_web_sm', disable=None, memoize=True)[source]¶ Initialize self. See help(type(self)) for accurate signature.
- Return type
None
Methods
__init__
(name, f[, resources, pre, …])Initialize self.