Rest Emitter

The Rest emitter can be used to push metadata to DataHub.

DataHubRestEmitter

class datahub.emitter.rest_emitter.DataHubRestEmitter(gms_server, token = None, timeout_sec = None, connect_timeout_sec = None, read_timeout_sec = None, retry_status_codes = None, retry_methods = None, retry_max_times = None, extra_headers = None, ca_certificate_path = None, client_certificate_path = None, disable_ssl_verification = False, openapi_ingestion = None, client_mode = None, datahub_component = None, server_config_refresh_interval = None)

Bases: Closeable, Emitter

• Parameters:
  • gms_server (str)
  • token (Optional[str])
  • timeout_sec (Optional[float])
  • connect_timeout_sec (Optional[float])
  • read_timeout_sec (Optional[float])
  • retry_status_codes (Optional[List[int]])
  • retry_methods (Optional[List[str]])
  • retry_max_times (Optional[int])
  • extra_headers (Optional[Dict[str, str]])
  • ca_certificate_path (Optional[str])
  • client_certificate_path (Optional[str])
  • disable_ssl_verification (bool)
  • openapi_ingestion (Optional[bool])
  • client_mode (Optional[ClientMode])
  • datahub_component (Optional[str])
  • server_config_refresh_interval (Optional[int])

close()

• Return type:None

emit(item, callback=None, emit_mode=EmitMode.SYNC_PRIMARY)

• Parameters:
  • item (Union[MetadataChangeEventClass, MetadataChangeProposalClass, MetadataChangeProposalWrapper, UsageAggregationClass]) –
  • callback (Optional[Callable[[Exception, str], None]])
  • emit_mode (EmitMode) –
• Return type:None

emit_mce(mce)

• Parameters:mce (MetadataChangeEventClass) –
• Return type:None

emit_mcp(mcp, async_flag=None, emit_mode=EmitMode.SYNC_PRIMARY, wait_timeout=datetime.timedelta(seconds=3600))

• Parameters:
  • mcp (Union[MetadataChangeProposalClass, MetadataChangeProposalWrapper]) –
  • async_flag (Optional[bool])
  • emit_mode (EmitMode) –
  • wait_timeout (Optional[timedelta])
• Return type:None

emit_mcps(mcps, emit_mode=EmitMode.SYNC_PRIMARY, wait_timeout=datetime.timedelta(seconds=3600))

• Parameters:
  • mcps (Sequence[Union[MetadataChangeProposalClass, MetadataChangeProposalWrapper]]) –
  • emit_mode (EmitMode) –
  • wait_timeout (Optional[timedelta])
• Return type:int

emit_usage(usageStats)

• Parameters:usageStats (UsageAggregationClass) –
• Return type:None

fetch_server_config()

Fetch configuration from the server if not already loaded.

• Return type:RestServiceConfig
• Returns: The configuration dictionary
• Raises:ConfigurationError – If there’s an error fetching or validating the configuration

flush()

• Return type:None

get_server_config()

• Return type:dict

invalidate_config_cache()

Manually invalidate the configuration cache.

• Return type:None

property server_config : RestServiceConfig

test_connection()

• Return type:None

to_graph()

• Return type:DataHubGraph

DatahubRestEmitter

datahub.emitter.rest_emitter.DatahubRestEmitter()

alias of DataHubRestEmitter

EmitMode

class datahub.emitter.rest_emitter.EmitMode(value)

Bases: ConfigEnum

An enumeration.

ASYNC = 'ASYNC'

ASYNC_WAIT = 'ASYNC_WAIT'

SYNC_PRIMARY = 'SYNC_PRIMARY'

SYNC_WAIT = 'SYNC_WAIT'

RequestsSessionConfig

class datahub.emitter.rest_emitter.RequestsSessionConfig(**data)

Bases: ConfigModel

• Parameters:
  • data (Any)
  • timeout (float | Tuple [float,float] | None)
  • retry_status_codes (List [int])
  • retry_methods (List [str])
  • retry_max_times (int)
  • extra_headers (Dict [str,str])
  • ca_certificate_path (str | None)
  • client_certificate_path (str | None)
  • disable_ssl_verification (bool)
  • client_mode (ClientMode | None)
  • datahub_component (str | None)

build_session()

• Return type:Session

ca_certificate_path : Optional[str]

client_certificate_path : Optional[str]

client_mode : Optional[ClientMode]

datahub_component : Optional[str]

disable_ssl_verification : bool

extra_headers : Dict[str, str]

classmethod get_client_mode_from_session(session)

Extract the ClientMode enum from a requests Session by checking the headers.

• Parameters:session (Session) – The requests.Session object to check
• Return type:Optional[ClientMode]
• Returns: The corresponding ClientMode enum value if found, None otherwise

model_config : ClassVar[ConfigDict] = {'_schema_extra': <function ConfigModel.Config._schema_extra>, 'extra': 'forbid', 'ignored_types': (<class 'cached_property.cached_property'>,), 'json_schema_extra': <function ConfigModel.Config._schema_extra>}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

retry_max_times : int

retry_methods : List[str]

retry_status_codes : List[int]

timeout : Union[float, Tuple[float, float], None]

RestSinkEndpoint

class datahub.emitter.rest_emitter.RestSinkEndpoint(value)

Bases: ConfigEnum

An enumeration.

OPENAPI = 'OPENAPI'

RESTLI = 'RESTLI'

preserve_unicode_escapes

datahub.emitter.rest_emitter.preserve_unicode_escapes(obj)

Recursively convert unicode characters back to escape sequences

• Parameters:obj (Any)
• Return type:Any