TensorFlow 模型分析

在 TensorFlow.org 上檢視 在 Google Colab 中執行 在 GitHub 上檢視原始碼 下載筆記本

TensorFlow Extended (TFX) 主要元件範例

TensorFlow 模型分析 (TFMA) 是一個程式庫，用於針對不同的資料切片執行模型評估。TFMA 使用 Apache Beam，以分散式方式對大量資料執行運算。

這個 Colab 筆記本範例說明如何使用 TFMA 來調查及視覺化模型在資料集特性方面的效能。我們會使用先前訓練的模型，現在您可以試用結果了！我們訓練的模型適用於 Chicago Taxi Example，這個範例使用芝加哥市發布的 Taxi Trips 資料集。在 BigQuery UI 中探索完整資料集。

身為模型建立者和開發人員，請思考如何使用這個資料，以及模型預測可能帶來的潛在益處和危害。像這樣的模型可能會強化社會偏見和差距。特徵與您想解決的問題相關嗎？還是會造成偏見？如需詳細資訊，請參閱ML 公平性。

資料集中的資料欄如下：

pickup_community_areafaretrip_start_month
trip_start_hourtrip_start_daytrip_start_timestamp
pickup_latitudepickup_longitudedropoff_latitude
dropoff_longitudetrip_milespickup_census_tract
dropoff_census_tractpayment_typecompany
trip_secondsdropoff_community_areatips

安裝 Jupyter 擴充功能

jupyter nbextension enable --py widgetsnbextension --sys-prefix 
jupyter nbextension install --py --symlink tensorflow_model_analysis --sys-prefix 
jupyter nbextension enable --py tensorflow_model_analysis --sys-prefix

安裝 TensorFlow 模型分析 (TFMA)

這會提取所有依附元件，而且需要一分鐘的時間。

# Upgrade pip to the latest, and install TFMA.
pip install -U pip
pip install tensorflow-model-analysis

現在您必須重新啟動執行階段，才能執行下方的儲存格。

# This setup was tested with TF 2.10 and TFMA 0.41 (using colab), but it should
# also work with the latest release.
import sys

# Confirm that we're using Python 3
assert sys.version_info.major==3, 'This notebook must be run using Python 3.'

import tensorflow as tf
print('TF version: {}'.format(tf.__version__))
import apache_beam as beam
print('Beam version: {}'.format(beam.__version__))
import tensorflow_model_analysis as tfma
print('TFMA version: {}'.format(tfma.__version__))

2024-04-30 10:58:28.448131: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-04-30 10:58:28.448179: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-04-30 10:58:28.449816: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
TF version: 2.15.1
Beam version: 2.55.1
TFMA version: 0.46.0

載入檔案

我們會下載一個 tar 檔案，其中包含我們需要的所有內容。其中包括

  • 訓練和評估資料集
  • 資料結構定義
  • 訓練和服務儲存模型 (keras 和 estimator) 以及評估儲存模型 (estimator)。
# Download the tar file from GCP and extract it
import io, os, tempfile
TAR_NAME = 'saved_models-2.2'
BASE_DIR = tempfile.mkdtemp()
DATA_DIR = os.path.join(BASE_DIR, TAR_NAME, 'data')
MODELS_DIR = os.path.join(BASE_DIR, TAR_NAME, 'models')
SCHEMA = os.path.join(BASE_DIR, TAR_NAME, 'schema.pbtxt')
OUTPUT_DIR = os.path.join(BASE_DIR, 'output')

!curl -O https://storage.googleapis.com/artifacts.tfx-oss-public.appspot.com/datasets/{TAR_NAME}.tar
!tar xf {TAR_NAME}.tar
!mv {TAR_NAME} {BASE_DIR}
!rm {TAR_NAME}.tar

print("Here's what we downloaded:")
!ls -R {BASE_DIR}

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6800k  100 6800k    0     0  26.9M      0 --:--:-- --:--:-- --:--:-- 26.9M
Here's what we downloaded:
/tmpfs/tmp/tmpo4hj24ht:
saved_models-2.2

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2:
data  models  schema.pbtxt

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/data:
eval  train

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/data/eval:
data.csv

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/data/train:
data.csv

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models:
estimator  keras

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator:
eval_model_dir  serving_model_dir

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/eval_model_dir:
1591221811

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/eval_model_dir/1591221811:
saved_model.pb  tmp.pbtxt  variables

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/eval_model_dir/1591221811/variables:
variables.data-00000-of-00001  variables.index

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir:
checkpoint
eval_chicago-taxi-eval
events.out.tfevents.1591221780.my-pipeline-b57vp-237544850
export
graph.pbtxt
model.ckpt-100.data-00000-of-00001
model.ckpt-100.index
model.ckpt-100.meta

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir/eval_chicago-taxi-eval:
events.out.tfevents.1591221799.my-pipeline-b57vp-237544850

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir/export:
chicago-taxi

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir/export/chicago-taxi:
1591221801

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir/export/chicago-taxi/1591221801:
saved_model.pb  variables

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/serving_model_dir/export/chicago-taxi/1591221801/variables:
variables.data-00000-of-00001  variables.index

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras:
0  1  2

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/0:
saved_model.pb  variables

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/0/variables:
variables.data-00000-of-00001  variables.index

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/1:
saved_model.pb  variables

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/1/variables:
variables.data-00000-of-00001  variables.index

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/2:
saved_model.pb  variables

/tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/keras/2/variables:
variables.data-00000-of-00001  variables.index

剖析結構定義

在我們下載的項目中，有一個適用於我們資料的結構定義，是由 TensorFlow Data Validation 建立的。現在就來剖析這個結構定義，以便搭配 TFMA 使用。

import tensorflow as tf
from google.protobuf import text_format
from tensorflow.python.lib.io import file_io
from tensorflow_metadata.proto.v0 import schema_pb2
from tensorflow.core.example import example_pb2

schema = schema_pb2.Schema()
contents = file_io.read_file_to_string(SCHEMA)
schema = text_format.Parse(contents, schema)

使用結構定義建立 TFRecords

我們需要讓 TFMA 存取我們的資料集，因此讓我們建立一個 TFRecords 檔案。我們可以使用結構定義來建立，因為結構定義為我們提供了每個特徵的正確類型。

import csv

datafile = os.path.join(DATA_DIR, 'eval', 'data.csv')
reader = csv.DictReader(open(datafile, 'r'))
examples = []
for line in reader:
  example = example_pb2.Example()
  for feature in schema.feature:
    key = feature.name
    if feature.type == schema_pb2.FLOAT:
      example.features.feature[key].float_list.value[:] = (
          [float(line[key])] if len(line[key]) > 0 else [])
    elif feature.type == schema_pb2.INT:
      example.features.feature[key].int64_list.value[:] = (
          [int(line[key])] if len(line[key]) > 0 else [])
    elif feature.type == schema_pb2.BYTES:
      example.features.feature[key].bytes_list.value[:] = (
          [line[key].encode('utf8')] if len(line[key]) > 0 else [])
  # Add a new column 'big_tipper' that indicates if tips was > 20% of the fare. 
  # TODO(b/157064428): Remove after label transformation is supported for Keras.
  big_tipper = float(line['tips']) > float(line['fare']) * 0.2
  example.features.feature['big_tipper'].float_list.value[:] = [big_tipper]
  examples.append(example)

tfrecord_file = os.path.join(BASE_DIR, 'train_data.rio')
with tf.io.TFRecordWriter(tfrecord_file) as writer:
  for example in examples:
    writer.write(example.SerializeToString())

!ls {tfrecord_file}

/tmpfs/tmp/tmpo4hj24ht/train_data.rio

設定並執行 TFMA

TFMA 支援多種不同的模型類型，包括 TF keras 模型、以一般 TF2 簽名 API 為基礎的模型，以及 TF estimator 型模型。《入門》指南中列出完整支援的模型類型和任何限制。在這個範例中，我們將說明如何設定以 keras 為基礎的模型，以及儲存為 EvalSavedModel 的 estimator 型模型。如需其他設定範例，請參閱常見問題。

TFMA 支援計算訓練時使用的指標 (即內建指標)，以及在模型儲存後定義的指標，這些指標是 TFMA 設定設定的一部分。針對我們的 keras 設定，我們將示範如何以手動方式新增指標和圖表，做為我們設定的一部分 (如需支援的指標和圖表資訊，請參閱指標指南)。針對 estimator 設定，我們會使用與模型一併儲存的內建指標。我們的設定也包含一些切片規格，這些規格將在以下章節中詳細討論。

建立 tfma.EvalConfig 和 tfma.EvalSharedModel 後，我們就可以使用 tfma.run_model_analysis 執行 TFMA。這會建立 tfma.EvalResult，稍後我們可以使用這個結果來轉譯指標和圖表。

Keras

import tensorflow_model_analysis as tfma

# Setup tfma.EvalConfig settings
keras_eval_config = text_format.Parse("""
  ## Model information
  model_specs {
    # For keras (and serving models) we need to add a `label_key`.
    label_key: "big_tipper"
  }

  ## Post training metric information. These will be merged with any built-in
  ## metrics from training.
  metrics_specs {
    metrics { class_name: "ExampleCount" }
    metrics { class_name: "AUC" }
    metrics { class_name: "Precision" }
    metrics { class_name: "Recall" }
    metrics { class_name: "MeanPrediction" }
    metrics { class_name: "Calibration" }
    metrics { class_name: "CalibrationPlot" }
    metrics { class_name: "ConfusionMatrixPlot" }
    # ... add additional metrics and plots ...
  }

  ## Slicing information
  slicing_specs {}  # overall slice
  slicing_specs {
    feature_keys: ["trip_start_hour"]
  }
  slicing_specs {
    feature_keys: ["trip_start_day"]
  }
  slicing_specs {
    feature_values: {
      key: "trip_start_month"
      value: "1"
    }
  }
""", tfma.EvalConfig())

# Create a tfma.EvalSharedModel that points at our keras model.
keras_model_path = os.path.join(MODELS_DIR, 'keras', '2')
keras_eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=keras_model_path,
    eval_config=keras_eval_config)

keras_output_path = os.path.join(OUTPUT_DIR, 'keras')

# Run TFMA
keras_eval_result = tfma.run_model_analysis(
    eval_shared_model=keras_eval_shared_model,
    eval_config=keras_eval_config,
    data_location=tfrecord_file,
    output_path=keras_output_path)

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`

Estimator

import tensorflow_model_analysis as tfma

# Setup tfma.EvalConfig settings
estimator_eval_config = text_format.Parse("""
  ## Model information
  model_specs {
    # To use EvalSavedModel set `signature_name` to "eval".
    signature_name: "eval"
  }

  ## Post training metric information. These will be merged with any built-in
  ## metrics from training.
  metrics_specs {
    metrics { class_name: "ConfusionMatrixPlot" }
    # ... add additional metrics and plots ...
  }

  ## Slicing information
  slicing_specs {}  # overall slice
  slicing_specs {
    feature_keys: ["trip_start_hour"]
  }
  slicing_specs {
    feature_keys: ["trip_start_day"]
  }
  slicing_specs {
    feature_values: {
      key: "trip_start_month"
      value: "1"
    }
  }
""", tfma.EvalConfig())

# Create a tfma.EvalSharedModel that points at our eval saved model.
estimator_base_model_path = os.path.join(
    MODELS_DIR, 'estimator', 'eval_model_dir')
estimator_model_path = os.path.join(
    estimator_base_model_path, os.listdir(estimator_base_model_path)[0])
estimator_eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=estimator_model_path,
    eval_config=estimator_eval_config)

estimator_output_path = os.path.join(OUTPUT_DIR, 'estimator')

# Run TFMA
estimator_eval_result = tfma.run_model_analysis(
    eval_shared_model=estimator_eval_shared_model,
    eval_config=estimator_eval_config,
    data_location=tfrecord_file,
    output_path=estimator_output_path)

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:163: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.saved_model.load` instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/eval_saved_model/load.py:163: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.saved_model.load` instead.
INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/eval_model_dir/1591221811/variables/variables
INFO:tensorflow:Restoring parameters from /tmpfs/tmp/tmpo4hj24ht/saved_models-2.2/models/estimator/eval_model_dir/1591221811/variables/variables
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This API was designed for TensorFlow v1. See https://tensorflow.dev.org.tw/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
2024-04-30 10:58:52.926791: W tensorflow/core/common_runtime/type_inference.cc:339] Type inference failed. This indicates an invalid graph that escaped type checking. Error message: INVALID_ARGUMENT: expected compatible input types, but input 1:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_INT64
    }
  }
}
 is neither a subtype nor a supertype of the combined inputs preceding it:
type_id: TFT_OPTIONAL
args {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_TENSOR
    args {
      type_id: TFT_INT32
    }
  }
}

    for Tuple type infernce function 0
    while inferring type of node 'dnn/zero_fraction/cond/output/_9'
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/eval_saved_model/graph_ref.py:184: get_tensor_from_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This API was designed for TensorFlow v1. See https://tensorflow.dev.org.tw/guide/migrate for instructions on how to migrate your code to TensorFlow v2.
2024-04-30 10:58:53.077553: W tensorflow/c/c_api.cc:305] Operation '{name:'head/metrics/true_positives_1/Assign' id:674 op device:{requested: '', assigned: ''} def:{ { {node head/metrics/true_positives_1/Assign} } = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](head/metrics/true_positives_1, head/metrics/true_positives_1/Initializer/zeros)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2024-04-30 10:58:53.204776: W tensorflow/c/c_api.cc:305] Operation '{name:'head/metrics/true_positives_1/Assign' id:674 op device:{requested: '', assigned: ''} def:{ { {node head/metrics/true_positives_1/Assign} } = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](head/metrics/true_positives_1, head/metrics/true_positives_1/Initializer/zeros)} }' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)

視覺化指標和圖表

現在我們已執行評估，讓我們來看看使用 TFMA 的視覺化結果。在以下範例中，我們將視覺化在 keras 模型上執行評估所得的結果。如要檢視以 estimator 為基礎的模型，請更新 eval_result_path 以指向我們的 estimator_output_path 變數。

eval_result_path = keras_output_path
# eval_result_path = estimator_output_path

eval_result = keras_eval_result
# eval_result = estimator_eval_result

轉譯指標

TFMA 在 tfma.experimental.dataframe 中提供 dataframe API，可將具體化的輸出內容載入為 Pandas DataFrames。如要檢視指標，您可以使用 metrics_as_dataframes(tfma.load_metrics(eval_path))，這會傳回一個物件，其中可能包含數個 DataFrame，每個 DataFrame 各適用於一種指標值類型 (double_value、confusion_matrix_at_thresholds、bytes_value 和 array_value)。填入的特定 DataFrame 取決於評估結果。在這裡，我們以 double_value DataFrame 為例。

import tensorflow_model_analysis.experimental.dataframe as tfma_dataframe
dfs = tfma_dataframe.metrics_as_dataframes(
  tfma.load_metrics(eval_result_path))

display(dfs.double_value.head())

每個 DataFrame 都有一個資料欄多重索引，最上層的資料欄為：slices、metric_keys 和 metric_values。每個群組的確切資料欄可能會根據酬載而變更。我們可以使用 DataFrame.columns API 檢查所有多重索引資料欄。舉例來說，切片資料欄為「Overall」、「trip_start_day」、「trip_start_hour」和「trip_start_month」，這是由 eval_config 中的 slicing_specs 設定的。

print(dfs.double_value.columns)

MultiIndex([(       'slices',  'trip_start_hour'),
            (       'slices',          'Overall'),
            (       'slices',   'trip_start_day'),
            (       'slices', 'trip_start_month'),
            (  'metric_keys',             'name'),
            (  'metric_keys',       'model_name'),
            (  'metric_keys',      'output_name'),
            (  'metric_keys', 'example_weighted'),
            (  'metric_keys',          'is_diff'),
            ('metric_values',     'double_value')],
           )

自動樞紐分析

DataFrame 的設計相當詳細，因此酬載不會遺失任何資訊。不過，有時為了直接使用，我們可能會想要以更精簡但有損耗的形式整理資訊：將切片做為資料列，並將指標做為資料欄。TFMA 提供 auto_pivot API 以達成此目的。這個公用程式會樞紐分析 metric_keys 內所有非唯一資料欄，並依預設將所有切片濃縮成一個 stringified_slices 資料欄。

tfma_dataframe.auto_pivot(dfs.double_value).head()

篩選切片

由於輸出內容是 DataFrame，因此任何原生 DataFrame API 都可用於切片和剖析 DataFrame。舉例來說，如果我們只對 trip_start_hour 為 1、3、5、7 的切片感興趣，而對 trip_start_day 的切片不感興趣，則可以使用 DataFrame 的 .loc 篩選邏輯。同樣地，我們使用 auto_pivot 函式，在執行篩選後，以切片與指標檢視重新整理 DataFrame。

df_double = dfs.double_value
df_filtered = (df_double
  .loc[df_double.slices.trip_start_hour.isin([1,3,5,7])]
)
display(tfma_dataframe.auto_pivot(df_filtered))

依指標值排序

我們也可以依指標值排序切片。舉例來說，我們示範如何依遞增 AUC 排序上述 DataFrame 中的切片，以便找出效能不佳的切片。這需要兩個步驟：自動樞紐分析，讓切片以資料列表示，而資料欄為指標；然後依 AUC 資料欄排序樞紐分析的 DataFrame。

# Pivoted table sorted by AUC in ascending order.
df_sorted = (
    tfma_dataframe.auto_pivot(df_double)
    .sort_values(by='auc', ascending=True)
    )
display(df_sorted.head())

轉譯圖表

新增至 tfma.EvalConfig 做為訓練後 metric_specs 的任何圖表，都可以使用 tfma.view.render_plot 顯示。

與指標一樣，圖表可以依切片檢視。與指標不同的是，只能顯示特定切片值的圖表，因此必須使用 tfma.SlicingSpec，而且必須同時指定切片特徵名稱和值。如果未提供任何切片，則會使用 Overall 切片的圖表。

在以下範例中，我們會顯示針對 trip_start_hour:1 切片計算的 CalibrationPlot 和 ConfusionMatrixPlot 圖表。

tfma.view.render_plot(
    eval_result,
    tfma.SlicingSpec(feature_values={'trip_start_hour': '1'}))

PlotViewer(config={'sliceName': 'trip_start_hour:1', 'metricKeys': {'calibrationPlot': {'metricName': 'calibra…

追蹤模型效能在不同時間的變化

您的訓練資料集將用於訓練模型，而且希望能夠代表您的測試資料集，以及將傳送至您在生產環境中模型的資料。不過，雖然推論要求中的資料可能與您的訓練資料保持不變，但在許多情況下，資料會開始產生足夠的變化，以致於模型的效能會隨之改變。

這表示您需要持續監控和評估模型的效能，以便瞭解變化並做出反應。讓我們來看看 TFMA 如何提供協助。

讓我們載入 3 個不同的模型執行，並使用 TFMA 透過 render_time_series 比較這些執行結果。

# Note this re-uses the EvalConfig from the keras setup.

# Run eval on each saved model
output_paths = []
for i in range(3):
  # Create a tfma.EvalSharedModel that points at our saved model.
  eval_shared_model = tfma.default_eval_shared_model(
      eval_saved_model_path=os.path.join(MODELS_DIR, 'keras', str(i)),
      eval_config=keras_eval_config)

  output_path = os.path.join(OUTPUT_DIR, 'time_series', str(i))
  output_paths.append(output_path)

  # Run TFMA
  tfma.run_model_analysis(eval_shared_model=eval_shared_model,
                          eval_config=keras_eval_config,
                          data_location=tfrecord_file,
                          output_path=output_path)

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives

首先，我們假設我們昨天訓練並部署了模型，現在我們想要查看模型在今天傳入的新資料上的效能。視覺化作業會先顯示 AUC。在 UI 中，您可以

  • 使用「新增指標序列」選單新增其他指標。
  • 按一下 x 關閉不想要的圖表
  • 將滑鼠游標懸停在資料點 (圖表中的線段末端) 上以取得更多詳細資料
eval_results_from_disk = tfma.load_eval_results(output_paths[:2])

tfma.view.render_time_series(eval_results_from_disk)

TimeSeriesViewer(config={'isModelCentric': True}, data=[{'metrics': {'': {'': {'example_count': {'doubleValue'…

現在我們假設又過了一天，我們想要查看模型在今天傳入的新資料上的效能，與前兩天相比如何

eval_results_from_disk = tfma.load_eval_results(output_paths)

tfma.view.render_time_series(eval_results_from_disk)

TimeSeriesViewer(config={'isModelCentric': True}, data=[{'metrics': {'': {'': {'example_count': {'doubleValue'…

模型驗證

TFMA 可以設定為同時評估多個模型。一般而言，這麼做是為了比較新模型與基準線 (例如目前服務中的模型)，以判斷指標 (例如 AUC 等) 的效能差異相對於基準線為何。設定 門檻後，TFMA 會產生 tfma.ValidationResult 記錄，指出效能是否符合預期。

讓我們重新設定 keras 評估，以比較兩個模型：候選模型和基準線模型。我們也會透過在 AUC 指標上設定 tmfa.MetricThreshold，驗證候選模型的效能是否符合基準線模型。

# Setup tfma.EvalConfig setting
eval_config_with_thresholds = text_format.Parse("""
  ## Model information
  model_specs {
    name: "candidate"
    # For keras we need to add a `label_key`.
    label_key: "big_tipper"
  }
  model_specs {
    name: "baseline"
    # For keras we need to add a `label_key`.
    label_key: "big_tipper"
    is_baseline: true
  }

  ## Post training metric information
  metrics_specs {
    metrics { class_name: "ExampleCount" }
    metrics { class_name: "BinaryAccuracy" }
    metrics { class_name: "BinaryCrossentropy" }
    metrics {
      class_name: "AUC"
      threshold {
        # Ensure that AUC is always > 0.9
        value_threshold {
          lower_bound { value: 0.9 }
        }
        # Ensure that AUC does not drop by more than a small epsilon
        # e.g. (candidate - baseline) > -1e-10 or candidate > baseline - 1e-10
        change_threshold {
          direction: HIGHER_IS_BETTER
          absolute { value: -1e-10 }
        }
      }
    }
    metrics { class_name: "AUCPrecisionRecall" }
    metrics { class_name: "Precision" }
    metrics { class_name: "Recall" }
    metrics { class_name: "MeanLabel" }
    metrics { class_name: "MeanPrediction" }
    metrics { class_name: "Calibration" }
    metrics { class_name: "CalibrationPlot" }
    metrics { class_name: "ConfusionMatrixPlot" }
    # ... add additional metrics and plots ...
  }

  ## Slicing information
  slicing_specs {}  # overall slice
  slicing_specs {
    feature_keys: ["trip_start_hour"]
  }
  slicing_specs {
    feature_keys: ["trip_start_day"]
  }
  slicing_specs {
    feature_keys: ["trip_start_month"]
  }
  slicing_specs {
    feature_keys: ["trip_start_hour", "trip_start_day"]
  }
""", tfma.EvalConfig())

# Create tfma.EvalSharedModels that point at our keras models.
candidate_model_path = os.path.join(MODELS_DIR, 'keras', '2')
baseline_model_path = os.path.join(MODELS_DIR, 'keras', '1')
eval_shared_models = [
  tfma.default_eval_shared_model(
      model_name=tfma.CANDIDATE_KEY,
      eval_saved_model_path=candidate_model_path,
      eval_config=eval_config_with_thresholds),
  tfma.default_eval_shared_model(
      model_name=tfma.BASELINE_KEY,
      eval_saved_model_path=baseline_model_path,
      eval_config=eval_config_with_thresholds),
]

validation_output_path = os.path.join(OUTPUT_DIR, 'validation')

# Run TFMA
eval_result_with_validation = tfma.run_model_analysis(
    eval_shared_models,
    eval_config=eval_config_with_thresholds,
    data_location=tfrecord_file,
    output_path=validation_output_path)

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:absl:Tensorflow version (2.15.1) found. Note that TFMA support for TF 2.0 is currently in beta
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/confusion_matrix_metrics.py:528: RuntimeWarning: invalid value encountered in divide
  prec_slope = dtp / np.maximum(dp, 0)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/confusion_matrix_metrics.py:532: RuntimeWarning: divide by zero encountered in divide
  p[:num_thresholds - 1] / np.maximum(p[1:], 0), np.ones_like(p[1:]))
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/confusion_matrix_metrics.py:532: RuntimeWarning: invalid value encountered in divide
  p[:num_thresholds - 1] / np.maximum(p[1:], 0), np.ones_like(p[1:]))
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:152: RuntimeWarning: invalid value encountered in divide
  f1 = 2 * precision * recall / (precision + recall)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/binary_confusion_matrices.py:155: RuntimeWarning: invalid value encountered in divide
  false_omission_rate = fn / predicated_negatives
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/confusion_matrix_metrics.py:539: RuntimeWarning: invalid value encountered in divide
  recall = tp / (tp + fn)
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/metrics/confusion_matrix_metrics.py:534: RuntimeWarning: invalid value encountered in divide
  prec_slope * (dtp + intercept * np.log(safe_p_ratio)) /

當針對基準線執行一個或多個模型的評估時，TFMA 會自動為評估期間計算的所有指標新增差異指標。這些指標會以對應的指標命名，但在指標名稱後方附加 _diff。

讓我們來看看我們的執行產生的指標

tfma.view.render_time_series(eval_result_with_validation)

TimeSeriesViewer(config={'isModelCentric': True}, data=[{'metrics': {'': {'': {'binary_crossentropy': {'double…

現在讓我們來看看驗證檢查的輸出內容。如要檢視驗證結果，我們可以使用 tfma.load_validator_result。在我們的範例中，驗證失敗，因為 AUC 低於門檻。

validation_result = tfma.load_validation_result(validation_output_path)
print(validation_result.validation_ok)

False