How do i get the features from yamnet in flutter using tflite?

33 Views Asked by At

I am building a flutter app, which uses the tflite yamnet model, to extract embeddings and spectrograms from audio. So far i have the following code in order to set the input and identify the outputs:

final interpreter = await Interpreter.fromAsset('assets/1.tflite');
interpreter.resizeInputTensor(0, singleSample.shape);
inputTensors = interpreter.getInputTensors();
Tensor inputTensor = inputTensors[0];

outputTensors = interpreter.getOutputTensors();
scores = outputTensors[0];
embeddings = outputTensors[1];
spectrogram = outputTensors[2];

print('Input tensor: $inputTensor');
print('Scores tensor: $scores');
print('Embeddings tensor: $embeddings');
print('Spectrogram tensor $spectrogram');

Which prints out the following four tensors:

Input tensor: Tensor{_tensor: Pointer: address=0xc6655640, name: waveform, >type: float32, shape: [8000, 1], data: 32000}

Scores tensor: Tensor{_tensor: Pointer: address=0xc6658900, name: Identity, >type: float32, shape: [1, 521], data: 2084}

Embeddings tensor: Tensor{_tensor: Pointer: address=0xc6658880, name: >Identity_1, type: float32, shape: [1, 1024], data: 4096}

Spectrogram tensor Tensor{_tensor: Pointer: address=0xc66579c0, name: >Identity_2, type: float32, shape: [1, 64], data: 256}

My problem arrives when i try to run the model and get the three outputs. I've tried the following:

  final input = List.generate(8000, (_) => List.filled(1, 0));
  final output = {0: scores, 1: embeddings, 2: spectrogram};
  interpreter.runForMultipleInputs(input, output);

Which gives error:

E/flutter (11660): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: RangeError (index): Invalid value: Only valid value is 0: 1 E/flutter (11660): #0 _Array.[] (dart:core-patch/array.dart:10:36) E/flutter (11660): #1 ListMixin.elementAt (dart:collection/list.dart:78:33) E/flutter (11660): #2 Interpreter.runInference interpreter.dart:196 E/flutter (11660): #3 Interpreter.runForMultipleInputs interpreter.dart:180 E/flutter (11660): #4 get_yamnet_features_single_sample yamnet_features.dart:31

And:

  final input = List.generate(8000, (_) => List.filled(1, 0));
  final output = [scores, embeddings, spectrogram];
  interpreter.run(input, output);

Which gives error:

E/tflite (11660): tensorflow/lite/kernels/pad.cc:79 SizeOfDimension(op_context->paddings, 0) != op_context->dims (1 != 2) E/tflite (11660): Node number 15 (PAD) failed to invoke. E/flutter (11660): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Bad state: failed precondition E/flutter (11660): #0 checkState check.dart:74 E/flutter (11660): #1 Interpreter.invoke interpreter.dart:164 E/flutter (11660): #2 Interpreter.runInference interpreter.dart:214 E/flutter (11660): #3 Interpreter.runForMultipleInputs interpreter.dart:180 E/flutter (11660): #4 Interpreter.run interpreter.dart:172 E/flutter (11660): #5 get_yamnet_features_single_sample yamnet_features.dart:31

I am able to do it in python, but in flutter it seems like i need to allocate the tensors beforehand. So can someone help me understand how to format three output tensors, with scores, embeddings and spectrograms?

0

There are 0 best solutions below