[CPU] Enable DT for matmul with float/integer-mixed inputs

Enable DT for the matmul with float/integer-mixed LHS and RHS but the result is float.
This change also pulls the element type checks into findMmt4dUkernelType so we can make sure not to set encoding when ukernel is enabled but not supported; otherwise currently they will fallback to scalar loops. This is a temporary restriction and #15784 should remove it.
#15760

I tested with an f32xi8->f32 matmul similar to the original issue:
module {
  func.func @matmul_f32i8f32(%arg0: tensor<1x256xf32>, %arg1: tensor<256x512xi8>) -> tensor<1x512xf32> {
    %cst = arith.constant 0.000000e+00 : f32
    %0 = tensor.empty() : tensor<1x512xf32>
    %2 = linalg.fill ins(%cst : f32) outs(%0 : tensor<1x512xf32>) -> tensor<1x512xf32>
    %1 = linalg.matmul ins(%arg0, %arg1 : tensor<1x256xf32>, tensor<256x512xi8>) outs(%2 : tensor<1x512xf32>) -> tensor<1x512xf32>
    return %1 : tensor<1x512xf32>
  }
}

It looks like codegen can generate reasonable code with the current DT tile sizes:

x86_64 AVX512f: https://gist.github.com/pzread/702be524a251c4a2d6dc6f6a1718a881#file-avx512f_1x256x512xf32f8f32-s

Inner tile size NxMxK 1x128x1 is chosen

ARM64 Pixel8 CPU features: https://gist.github.com/pzread/702be524a251c4a2d6dc6f6a1718a881#file-arm64_pixel8_1x256x512xi8f32f32-s
Inner tile size NxMxK 1x8x1 is chosen

In both case fmla/vfmadd are generated. Although with AVX512 we generate quite amount of instructions to convert 128xi8 into 128xf32.
@dcaballe would you mind check if those instructions make sense?

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 looks scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump?
Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 is scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump? Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

The tile sizes on my AArch64 machine are actually 1x16x1. The "static" config is 8x16x1, but 8 is replaced with 1 based on the shapes of the inputs:

https://github.com/openxla/iree/blob/e799ae93e6ee92eb01245d8575eea4690f0c4735/compiler/src/iree/compiler/Codegen/LLVMCPU/KernelDispatch.cpp#L1096

Now, looking at assembly there's scope for improvement, but looks like an entirely separate issue :) (well, I should take a closer look)

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 is scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump? Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

Here are the IR dump for x86 https://gist.github.com/pzread/df7a7dbb40027cb02188c3a45781fc24 and for ARM64 https://gist.github.com/pzread/a49a0e77c5c7e6b629dbb253db039059

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 is scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump? Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

The tile sizes on my AArch64 machine are actually 1x16x1. The "static" config is 8x16x1, but 8 is replaced with 1 based on the shapes of the inputs:

https://github.com/openxla/iree/blob/e799ae93e6ee92eb01245d8575eea4690f0c4735/compiler/src/iree/compiler/Codegen/LLVMCPU/KernelDispatch.cpp#L1096

Now, looking at assembly there's scope for improvement, but looks like an entirely separate issue :) (well, I should take a closer look)

I would like to clarify that the inner tile size 1x8x1 is defined for data-tiling and decided here:
https://github.com/openxla/iree/blob/15ca58e19ec76fab94c4aba8f75091c532282d51/compiler/src/iree/compiler/Codegen/Common/CPU/CPUMaterializeEncodingPass.cpp#L103-L116
It is a separate decision from the normal tile size selection logic in KernelDispatch.cpp. The mmt4d op has 6 dimensions and outer 3 dims use the tile sizes from KernelDispatch.cpp and the inner 3 dims use the tile sizes I mentioned above.

Apologies, I had DT disabled when checking 🤦🏻 .
So, looking at ASM, this looks very suboptimal:
	smov	w7, v2.b[0]
	smov	w19, v2.b[1]
	smov	w20, v2.b[4]
	smov	w21, v2.b[5]
	smov	w22, v2.b[2]
	scvtf	s3, w7
	smov	w7, v2.b[6]
	scvtf	s4, w19
	scvtf	s5, w20
	scvtf	s6, w21
	scvtf	s7, w22d

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 is scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump? Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

The tile sizes on my AArch64 machine are actually 1x16x1. The "static" config is 8x16x1, but 8 is replaced with 1 based on the shapes of the inputs:

https://github.com/openxla/iree/blob/e799ae93e6ee92eb01245d8575eea4690f0c4735/compiler/src/iree/compiler/Codegen/LLVMCPU/KernelDispatch.cpp#L1096

Now, looking at assembly there's scope for improvement, but looks like an entirely separate issue :) (well, I should take a closer look)

I would like to clarify that the inner tile size 1x8x1 is defined for data-tiling and decided here:
https://github.com/openxla/iree/blob/15ca58e19ec76fab94c4aba8f75091c532282d51/compiler/src/iree/compiler/Codegen/Common/CPU/CPUMaterializeEncodingPass.cpp#L103-L116
It is a separate decision from the normal tile size selection logic in KernelDispatch.cpp. The mmt4d op has 6 dimensions and outer 3 dims use the tile sizes from KernelDispatch.cpp and the inner 3 dims use the tile sizes I mentioned above.

Apologies, I had DT disabled 🤦🏻 .
So, this from your ASM dump is pretty bad:
	smov	w7, v2.b[0]
	smov	w19, v2.b[1]
	smov	w20, v2.b[4]
	smov	w21, v2.b[5]
	smov	w22, v2.b[2]
	scvtf	s3, w7
	smov	w7, v2.b[6]
	scvtf	s4, w19
	scvtf	s5, w20
	scvtf	s6, w21
	scvtf	s7, w22

This is without DT:
  zip1  v6.8b, v1.8b, v0.8b
  zip2  v1.8b, v1.8b, v0.8b
  zip1  v7.8b, v3.8b, v0.8b
  zip2  v3.8b, v3.8b, v0.8b
  shl v6.4h, v6.4h, #8
  shl v1.4h, v1.4h, #8
  sshr  v6.4h, v6.4h, #8
  shl v7.4h, v7.4h, #8
  shl v3.4h, v3.4h, #8
  sshr  v1.4h, v1.4h, #8
  sshll v6.4s, v6.4h, #0
  sshr  v7.4h, v7.4h, #8
  sshr  v3.4h, v3.4h, #8
  sshll v1.4s, v1.4h, #0
  scvtf v6.4s, v6.4s
  sshll v7.4s, v7.4h, #0
  sshll v3.4s, v3.4h, #0
  scvtf v1.4s, v1.4s
  scvtf v7.4s, v7.4s
  scvtf v3.4s, v3.4s
  fmla  v0.4s, v6.4s, v16.s[0]
  fmla  v4.4s, v1.4s, v16.s[0]
  fmla  v2.4s, v3.4s, v16.s[0]
  fmla  v5.4s, v7.4s, v16.s[0]

So, codegen looks much better without DT. Have you tried with SVE enabled? Worth trying given that you are testing on Pixel-8.

So, codegen looks much better without DT. Have you tried with SVE enabled? Worth trying given that you are testing on Pixel-8.

I tried the CPU features below but still got the similar suboptimal code
iree-compile --iree-hal-target-backends=llvm-cpu --iree-llvmcpu-target-triple=aarch64-none-linux-android34 --iree-llvmcpu-target-cpu-features="+v9a,+fullfp16,+fp-armv8,+neon,+aes,+sha2,+crc,+lse,+rdm,+complxnum,+rcpc,+sha3,+sm4,+dotprod,+fp16fml,+dit,+flagm,+ssbs,+sb,+sve2-aes,+sve2-bitperm,+sve2-sha3,+sve2-sm4,+altnzcv,+fptoint,+bf16,+i8mm,+bti" --mlir-disable-threading --iree-opt-data-tiling --iree-llvmcpu-enable-ukernels=none

.LBB2_8:
	ldr	d2, [x6, x4, lsl #3]
	smov	w7, v2.b[0]
	smov	w19, v2.b[1]
	smov	w20, v2.b[4]
	smov	w21, v2.b[5]
	smov	w22, v2.b[2]
	scvtf	s3, w7
	smov	w7, v2.b[6]
	scvtf	s4, w19
	scvtf	s5, w20
	scvtf	s6, w21
	scvtf	s7, w22
	smov	w19, v2.b[3]
	scvtf	s16, w7
	smov	w7, v2.b[7]
	mov	v3.s[1], v4.s[0]
	mov	v5.s[1], v6.s[0]
	scvtf	s2, w19
	scvtf	s4, w7
	mov	v3.s[2], v7.s[0]
	mov	v5.s[2], v16.s[0]
	mov	v3.s[3], v2.s[0]
	ldr	s2, [x10, x4, lsl #2]
	add	x4, x4, #1
	mov	v5.s[3], v4.s[0]
	cmp	x4, #256
	fmla	v0.4s, v3.4s, v2.s[0]
	fmla	v1.4s, v5.4s, v2.s[0]
	b.ne	.LBB2_8
	.loc	3 0 8
	add	x4, x17, x5, lsl #5
	.loc	3 21 8
	add	x2, x2, #1
	add	x3, x3, #2048
	cmp	x2, #32
	stp	q0, q1, [x4]
	b.ne	.LBB2_7
	add	x1, x1, x13
	add	x0, x0, x15
	cmp	x1, #64
	b.lt	.LBB2_6
	b	.LBB2_3

So I checked the IR dump of ARM64, I'm wondering if the scalar code is due to the shape vector<8x1xi8> on the cast op:
%9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
%10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>

The related snippets:
// -----// IR Dump After OptimizeVectorTransfer (iree-codegen-optimize-vector-transfer) //----- //
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %c0_i8 = arith.constant 0 : i8
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %cst_0 = arith.constant 0.000000e+00 : f32
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_2 = memref.subview %subview_1[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_3 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.transfer_read %0[%c0, %arg3, %c0, %c0], %cst_0 {in_bounds = [true]} : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>, vector<1xf32>
          %8 = vector.transfer_read %subview_2[%arg2, %arg3, %c0], %c0_i8 {in_bounds = [true]} : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
          %10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>
          %11 = vector.broadcast %7 : vector<1xf32> to vector<1x1xf32>
          %12 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %13 = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>, affine_map<(d0, d1, d2) -> (d1, d2)>, affine_map<(d0, d1, d2) -> (d1)>], iterator_types = ["reduction", "parallel", "reduction"], kind = #vector.kind<add>} %11, %10, %12 : vector<1x1xf32>, vector<8x1xf32> into vector<8xf32>
          %14 = vector.broadcast %13 : vector<8xf32> to vector<1x8xf32>
          %15 = vector.broadcast %14 : vector<1x8xf32> to vector<1x1x8xf32>
          %16 = vector.broadcast %15 : vector<1x1x8xf32> to vector<1x1x1x8xf32>
          scf.yield %16 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_4 = memref.subview %subview_3[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.transfer_write %6, %subview_4[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
      }
    }
  }
  return
}

// -----// IR Dump After LLVMCPUVectorLowering (iree-llvmcpu-vector-lowering) //----- //
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %cst = arith.constant dense<0> : vector<8x1xi8>
  %cst_0 = arith.constant dense<0.000000e+00> : vector<1x8xf32>
  %cst_1 = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_2 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_4 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst_1) -> (vector<1x1x1x8xf32>) {
          %7 = vector.load %subview_3[%arg2, %arg3, %c0] : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = vector.extract_strided_slice %7 {offsets = [0], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %9 = vector.insert %8, %cst [0] : vector<1xi8> into vector<8x1xi8>
          %10 = vector.extract_strided_slice %7 {offsets = [1], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %11 = vector.insert %10, %9 [1] : vector<1xi8> into vector<8x1xi8>
          %12 = vector.extract_strided_slice %7 {offsets = [2], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %13 = vector.insert %12, %11 [2] : vector<1xi8> into vector<8x1xi8>
          %14 = vector.extract_strided_slice %7 {offsets = [3], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %15 = vector.insert %14, %13 [3] : vector<1xi8> into vector<8x1xi8>
          %16 = vector.extract_strided_slice %7 {offsets = [4], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %17 = vector.insert %16, %15 [4] : vector<1xi8> into vector<8x1xi8>
          %18 = vector.extract_strided_slice %7 {offsets = [5], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %19 = vector.insert %18, %17 [5] : vector<1xi8> into vector<8x1xi8>
          %20 = vector.extract_strided_slice %7 {offsets = [6], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %21 = vector.insert %20, %19 [6] : vector<1xi8> into vector<8x1xi8>
          %22 = vector.extract_strided_slice %7 {offsets = [7], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %23 = vector.insert %22, %21 [7] : vector<1xi8> into vector<8x1xi8>
          %24 = arith.sitofp %23 : vector<8x1xi8> to vector<8x1xf32>
          %25 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %26 = vector.extract %24[0, 0] : f32 from vector<8x1xf32>
          %27 = vector.insert %26, %cst_0 [0, 0] : f32 into vector<1x8xf32>
          %28 = vector.extract %24[1, 0] : f32 from vector<8x1xf32>
          %29 = vector.insert %28, %27 [0, 1] : f32 into vector<1x8xf32>
          %30 = vector.extract %24[2, 0] : f32 from vector<8x1xf32>
          %31 = vector.insert %30, %29 [0, 2] : f32 into vector<1x8xf32>
          %32 = vector.extract %24[3, 0] : f32 from vector<8x1xf32>
          %33 = vector.insert %32, %31 [0, 3] : f32 into vector<1x8xf32>
          %34 = vector.extract %24[4, 0] : f32 from vector<8x1xf32>
          %35 = vector.insert %34, %33 [0, 4] : f32 into vector<1x8xf32>
          %36 = vector.extract %24[5, 0] : f32 from vector<8x1xf32>
          %37 = vector.insert %36, %35 [0, 5] : f32 into vector<1x8xf32>
          %38 = vector.extract %24[6, 0] : f32 from vector<8x1xf32>
          %39 = vector.insert %38, %37 [0, 6] : f32 into vector<1x8xf32>
          %40 = vector.extract %24[7, 0] : f32 from vector<8x1xf32>
          %41 = vector.insert %40, %39 [0, 7] : f32 into vector<1x8xf32>
          %42 = vector.extract %41[0] : vector<8xf32> from vector<1x8xf32>
          %43 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %44 = vector.broadcast %43 : f32 to vector<8xf32>
          %45 = vector.fma %42, %44, %25 : vector<8xf32>
          %46 = vector.broadcast %45 : vector<8xf32> to vector<1x1x1x8xf32>
          scf.yield %46 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_5 = memref.subview %subview_4[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.store %6, %subview_5[%c0] : memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>, vector<8xf32>
      }
    }
  }
  return
}

With no data-tiling, the results of is much simpler:
// -----// IR Dump After LLVMCPUVectorLowering (iree-llvmcpu-vector-lowering) //----- //
func.func @matmul_f32i8f32_dispatch_0_matmul_1x512x256_f32xi8xf32() {
  %cst = arith.constant dense<0> : vector<1x16xi8>
  %cst_0 = arith.constant dense<0.000000e+00> : vector<1x16xf32>
  %c256 = arith.constant 256 : index
  %c1 = arith.constant 1 : index
  %c16 = arith.constant 16 : index
  %c64 = arith.constant 64 : index
  %c0 = arith.constant 0 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<256x512xi8, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<256x512xi8, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(2) type(storage_buffer) alignment(64) offset(%c0) : memref<1x512xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x512xf32, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 64)>()[%workgroup_id_x]
  %subview = memref.subview %2[0, %3] [1, 64] [1, 1] : memref<1x512xf32, #hal.descriptor_type<storage_buffer>> to memref<1x64xf32, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
  %subview_1 = memref.subview %1[0, %3] [256, 64] [1, 1] : memref<256x512xi8, #hal.descriptor_type<storage_buffer>> to memref<256x64xi8, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
  scf.for %arg0 = %c0 to %c64 step %c16 {
    %subview_2 = memref.subview %subview[0, %arg0] [1, 16] [1, 1] : memref<1x64xf32, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x16xf32, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
    %4 = scf.for %arg1 = %c0 to %c256 step %c1 iter_args(%arg2 = %cst_0) -> (vector<1x16xf32>) {
      %6 = vector.load %subview_1[%arg1, %arg0] : memref<256x64xi8, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xi8>
      %7 = vector.insert %6, %cst [0] : vector<16xi8> into vector<1x16xi8>
      %8 = arith.sitofp %7 : vector<1x16xi8> to vector<1x16xf32>
      %9 = vector.extract %8[0] : vector<16xf32> from vector<1x16xf32>
      %10 = memref.load %0[%c0, %arg1] : memref<1x256xf32, #hal.descriptor_type<storage_buffer>>
      %11 = vector.broadcast %10 : f32 to vector<16xf32>
      %12 = vector.extract %arg2[0] : vector<16xf32> from vector<1x16xf32>
      %13 = vector.fma %11, %9, %12 : vector<16xf32>
      %14 = vector.insert %13, %cst_0 [0] : vector<16xf32> into vector<1x16xf32>
      scf.yield %14 : vector<1x16xf32>
    }
    %5 = vector.extract %4[0] : vector<16xf32> from vector<1x16xf32>
    vector.store %5, %subview_2[%c0, %c0] : memref<1x16xf32, strided<[512, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xf32>
  }
  return
}

I tried the CPU features below but still got the similar suboptimal code

[side note] You are using quite a lot of target features. IIUC, we are targeting "basic" NEON/SVE here and looking at plain matmuls. So the additional extensions only contribute noise. You can simplify that to:
--iree-llvmcpu-target-cpu-features="+sve"

That's what I use for experiments (admittedly, my target triple is "aarch64-none-linux-gnu" rather than "aarch64-none-linux-android34").
As for your updated assembly generated with +sve, that doesn't contain any SVE intstructions (vector registers should switch from v to z). Which indicates that when SVE is enabled, data tiling is disabled. Any idea how to change that? I am very keen to see what happens. And help solve any issues there.

I checked the IR dump of ARM64 and wondering if the scalar code is due to the shape vector<8x1xi8> on the cast op:
%9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
%10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>

From LLVM perspective that's an array of 8 1-element vectors, so basically scalar code. So yeah, that's the culprit - good find!

I checked the IR dump of ARM64 and wondering if the scalar code is due to the shape vector<8x1xi8> on the cast op:
%9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
%10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>

Oh, no, unit dims again... this explains the problem for both x86 and aarch64 side. This is happening because this matmul has an embedded conversion within the same dispatch... and I guess it's the first time we look into an scenario like this.
Could we create a pass to remove unit dims from "shape-agnostic" ops (elementwise ops would be a good starting point). The pass could turn the previous arith.sitofp into:
 %9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
 %10 = vector.shape_cast %9 : vector<8x1xi8> to vector<8xi8>
 %11 = arith.sitofp %10 : vector<8xi8> to vector<8xf32>
 %12 = vector.shape_cast %11 : vector<8xf32> to vector<8x1xf32>

and then we could run shape_cast canonicalizations at the end of the pass to fold all the shape casts that can cancel out:
 %11 = arith.sitofp %10 : vector<8xi8> to vector<8xf32>
 %12 = vector.shape_cast %11 : vector<8xf32> to vector<8x1xf32>

Something like that should work in a general way and leave the unit dim only around the ops that really need them...
Just a suggestion.

I guess we can land this PR first. The unit removal could be a follow up.

Could we create a pass to remove unit dims from "shape-agnostic" ops (elementwise ops would be a good starting point).

@dcaballe https://github.com/llvm/llvm-project/pull/74817 :) And, specifically, this example.
I was actually planning to generalise my PR to elementwise OPs tomorrow - I am super glad that there's another example where this will be beneficial :)

@dcaballe llvm/llvm-project#74817 :) And, specifically, this example.
I was actually planning to generalise my PR to elementwise OPs tomorrow - I am super glad that there's another example where this will be beneficial :)

Awesome! Are you taking care of this then? Thanks in advance! :)

@dcaballe llvm/llvm-project#74817 :) And, specifically, this example.
I was actually planning to generalise my PR to elementwise OPs tomorrow - I am super glad that there's another example where this will be beneficial :)

Are you taking care of this then?

I would rather put in the bucket of PRs that "happen before you even know that you need them" 😂 . It should already work for this change, though I wasn't sure where to add these patterns - pointers are much appreciated.

@dcaballe llvm/llvm-project#74817 :) And, specifically, this example.
I was actually planning to generalise my PR to elementwise OPs tomorrow - I am super glad that there's another example where this will be beneficial :)

Are you taking care of this then?

I would rather put in the bucket of PRs that "happen before you even know that you need them" 😂 . It should already work for this change, though I wasn't sure where to add these patterns - pointers are much appreciated.

I guess we decompose vector.contract in LLVMCPUVectorLowering.cpp and at that point we reveal that we actually want to transpose the RHS vector<8x1xf32> into vector<8xf32> for vector.fma.
Especially I think the patterns here help cancel out the extra shape_cast and transpose (currently it works when there is no sitofp between the shape_cast and vector.contract). So maybe the pattern can be add here?
https://github.com/openxla/iree/blob/4c8f67d7f3e7621c0e38cac9783390191ee31c73/compiler/src/iree/compiler/Codegen/LLVMCPU/LLVMCPUVectorLowering.cpp#L120-L129
Below is the lowering process in LLVMCPUVectorLowering. We can see that after "--- After lowering high level vector ops to lower level vector ops ---", we reveal the pattern of shape_cast to vector<8x1xf32> -> sitofp -> transpose back to vector<1x8xf32>
--- After applying patterns for vector.gather Ops ---
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %c0_i8 = arith.constant 0 : i8
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %cst_0 = arith.constant 0.000000e+00 : f32
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_2 = memref.subview %subview_1[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_3 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.transfer_read %0[%c0, %arg3, %c0, %c0], %cst_0 {in_bounds = [true]} : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>, vector<1xf32>
          %8 = vector.transfer_read %subview_2[%arg2, %arg3, %c0], %c0_i8 {in_bounds = [true]} : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
          %10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>
          %11 = vector.broadcast %7 : vector<1xf32> to vector<1x1xf32>
          %12 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %13 = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>, affine_map<(d0, d1, d2) -> (d1, d2)>, affine_map<(d0, d1, d2) -> (d1)>], iterator_types = ["reduction", "parallel", "reduction"], kind = #vector.kind<add>} %11, %10, %12 : vector<1x1xf32>, vector<8x1xf32> into vector<8xf32>
          %14 = vector.broadcast %13 : vector<8xf32> to vector<1x8xf32>
          %15 = vector.broadcast %14 : vector<1x8xf32> to vector<1x1x8xf32>
          %16 = vector.broadcast %15 : vector<1x1x8xf32> to vector<1x1x1x8xf32>
          scf.yield %16 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_4 = memref.subview %subview_3[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.transfer_write %6, %subview_4[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
      }
    }
  }
  return
}

--- After lowering high level vector ops to lower level vector ops ---
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %c0_i8 = arith.constant 0 : i8
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_0 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %subview_0[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_2 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.transfer_read %subview_1[%arg2, %arg3, %c0], %c0_i8 {in_bounds = [true]} : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = vector.shape_cast %7 : vector<8xi8> to vector<8x1xi8>
          %9 = arith.sitofp %8 : vector<8x1xi8> to vector<8x1xf32>
          %10 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %11 = vector.transpose %9, [1, 0] : vector<8x1xf32> to vector<1x8xf32>
          %12 = vector.extract %11[0] : vector<8xf32> from vector<1x8xf32>
          %13 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %14 = vector.broadcast %13 : f32 to vector<8xf32>
          %15 = vector.fma %12, %14, %10 : vector<8xf32>
          %16 = vector.broadcast %15 : vector<8xf32> to vector<1x8xf32>
          %17 = vector.broadcast %16 : vector<1x8xf32> to vector<1x1x8xf32>
          %18 = vector.broadcast %17 : vector<1x1x8xf32> to vector<1x1x1x8xf32>
          scf.yield %18 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.transfer_write %6, %subview_3[%c0] {in_bounds = [true]} : vector<8xf32>, memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
      }
    }
  }
  return
}

--- After lowering vector transfers to SCF ---
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_0 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %subview_0[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_2 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.load %subview_1[%arg2, %arg3, %c0] : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = vector.shape_cast %7 : vector<8xi8> to vector<8x1xi8>
          %9 = arith.sitofp %8 : vector<8x1xi8> to vector<8x1xf32>
          %10 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %11 = vector.transpose %9, [1, 0] : vector<8x1xf32> to vector<1x8xf32>
          %12 = vector.extract %11[0] : vector<8xf32> from vector<1x8xf32>
          %13 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %14 = vector.broadcast %13 : f32 to vector<8xf32>
          %15 = vector.fma %12, %14, %10 : vector<8xf32>
          %16 = vector.broadcast %15 : vector<8xf32> to vector<1x1x1x8xf32>
          scf.yield %16 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.store %6, %subview_3[%c0] : memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>, vector<8xf32>
      }
    }
  }
  return
}

--- After lowering vector transpose ops ---
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_0 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %subview_0[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_2 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.load %subview_1[%arg2, %arg3, %c0] : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = vector.shape_cast %7 : vector<8xi8> to vector<8x1xi8>
          %9 = arith.sitofp %8 : vector<8x1xi8> to vector<8x1xf32>
          %10 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %11 = vector.shape_cast %9 : vector<8x1xf32> to vector<1x8xf32>
          %12 = vector.extract %11[0] : vector<8xf32> from vector<1x8xf32>
          %13 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %14 = vector.broadcast %13 : f32 to vector<8xf32>
          %15 = vector.fma %12, %14, %10 : vector<8xf32>
          %16 = vector.broadcast %15 : vector<8xf32> to vector<1x1x1x8xf32>
          scf.yield %16 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.store %6, %subview_3[%c0] : memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>, vector<8xf32>
      }
    }
  }
  return
}

// -----// IR Dump After LLVMCPUVectorLowering (iree-llvmcpu-vector-lowering) //----- //
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %cst = arith.constant dense<0> : vector<8x1xi8>
  %cst_0 = arith.constant dense<0.000000e+00> : vector<1x8xf32>
  %cst_1 = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_2 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_4 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst_1) -> (vector<1x1x1x8xf32>) {
          %7 = vector.load %subview_3[%arg2, %arg3, %c0] : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = vector.extract_strided_slice %7 {offsets = [0], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %9 = vector.insert %8, %cst [0] : vector<1xi8> into vector<8x1xi8>
          %10 = vector.extract_strided_slice %7 {offsets = [1], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %11 = vector.insert %10, %9 [1] : vector<1xi8> into vector<8x1xi8>
          %12 = vector.extract_strided_slice %7 {offsets = [2], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %13 = vector.insert %12, %11 [2] : vector<1xi8> into vector<8x1xi8>
          %14 = vector.extract_strided_slice %7 {offsets = [3], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %15 = vector.insert %14, %13 [3] : vector<1xi8> into vector<8x1xi8>
          %16 = vector.extract_strided_slice %7 {offsets = [4], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %17 = vector.insert %16, %15 [4] : vector<1xi8> into vector<8x1xi8>
          %18 = vector.extract_strided_slice %7 {offsets = [5], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %19 = vector.insert %18, %17 [5] : vector<1xi8> into vector<8x1xi8>
          %20 = vector.extract_strided_slice %7 {offsets = [6], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %21 = vector.insert %20, %19 [6] : vector<1xi8> into vector<8x1xi8>
          %22 = vector.extract_strided_slice %7 {offsets = [7], sizes = [1], strides = [1]} : vector<8xi8> to vector<1xi8>
          %23 = vector.insert %22, %21 [7] : vector<1xi8> into vector<8x1xi8>
          %24 = arith.sitofp %23 : vector<8x1xi8> to vector<8x1xf32>
          %25 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %26 = vector.extract %24[0, 0] : f32 from vector<8x1xf32>
          %27 = vector.insert %26, %cst_0 [0, 0] : f32 into vector<1x8xf32>
          %28 = vector.extract %24[1, 0] : f32 from vector<8x1xf32>
          %29 = vector.insert %28, %27 [0, 1] : f32 into vector<1x8xf32>
          %30 = vector.extract %24[2, 0] : f32 from vector<8x1xf32>
          %31 = vector.insert %30, %29 [0, 2] : f32 into vector<1x8xf32>
          %32 = vector.extract %24[3, 0] : f32 from vector<8x1xf32>
          %33 = vector.insert %32, %31 [0, 3] : f32 into vector<1x8xf32>
          %34 = vector.extract %24[4, 0] : f32 from vector<8x1xf32>
          %35 = vector.insert %34, %33 [0, 4] : f32 into vector<1x8xf32>
          %36 = vector.extract %24[5, 0] : f32 from vector<8x1xf32>
          %37 = vector.insert %36, %35 [0, 5] : f32 into vector<1x8xf32>
          %38 = vector.extract %24[6, 0] : f32 from vector<8x1xf32>
          %39 = vector.insert %38, %37 [0, 6] : f32 into vector<1x8xf32>
          %40 = vector.extract %24[7, 0] : f32 from vector<8x1xf32>
          %41 = vector.insert %40, %39 [0, 7] : f32 into vector<1x8xf32>
          %42 = vector.extract %41[0] : vector<8xf32> from vector<1x8xf32>
          %43 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %44 = vector.broadcast %43 : f32 to vector<8xf32>
          %45 = vector.fma %42, %44, %25 : vector<8xf32>
          %46 = vector.broadcast %45 : vector<8xf32> to vector<1x1x1x8xf32>
          scf.yield %46 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_5 = memref.subview %subview_4[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.store %6, %subview_5[%c0] : memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>, vector<8xf32>
      }
    }
  }
  return
}

There is no regression on OSS DT-only benchmarks but I'm seeing 44% regressions on our internal models.

In both case fmla/vfmadd are generated. Although with AVX512 we generate a quite amount of instructions to convert 128xi8 into 128xf32.

Oh oh... the i8 -> f32 is scalar... @qcolombet, it looks like an implementation gap in your i8 ->f16 fix? Would you have cycles to take a look?
The i8->f32 conversion on Pixel 8 doesn't look great either. Perhaps @banach-space can help here?
@pzread, could you please post the print after all dump? Also, regarding the tile sizes for Pixel8, we should use larger ones for the vecmat/matvec cases to preserve a similar unrolling factor than for regular matmuls.

The code doesn't look as bad as what was reported in the gist. My IREE repo is a few weeks old though, so let me rebase and see (I'm at ef0f1a405783c9cf673487a61fda8fca8e83e8c1).

Confirm the findings: what reaches the backend is already scalar code for the sitofp:
Before LLVM conversion:
    %48 = arith.sitofp %47 : vector<16x1xi8> to vector<16x1xf32>

After LLVM conversion:
    %105 = llvm.sitofp %89 : vector<1xi8> to vector<1xf32>
[...]
    %120 = llvm.sitofp %104 : vector<1xi8> to vector<1xf32>

If we can get rid of the unit dim all together, we could also avoid the dance for going from <16xi8> to <16x1xi8>.
I.e., this blob:
    %15 = vector.transfer_read %collapse_shape[%9, %11, %c0], %c0_i8 {in_bounds = [true]} : memref<4x256x16xi8, strided<[4096, 16, 1], offset: ?>>, vector<16xi8>
// first element
    %16 = vector.extract_strided_slice %15 {offsets = [0], sizes = [1], strides = [1]} : vector<16xi8> to vector<1xi8>
    %17 = vector.insert %16, %cst [0] : vector<1xi8> into vector<16x1xi8>
[...]
// 16-th element
    %46 = vector.extract_strided_slice %15 {offsets = [15], sizes = [1], strides = [1]} : vector<16xi8> to vector<1xi8>
    %47 = vector.insert %46, %45 [15] : vector<1xi8> into vector<16x1xi8>

We may be able to rebuild the vector code in the backend (maybe by repurposing the SLP vectorizer to look at chains going to insertelement?), but I won't hold my breath.

I tested a pattern to change sitofp into shape_cast -> sitofp -> shape_cast and I think the generated code looks better now:
// -----// IR Dump After LLVMCPUVectorLowering (iree-llvmcpu-vector-lowering) //----- //
func.func @matmul_f32i8f32_dispatch_2_mmt4d_1x64x256x1x8x1_f32xi8xf32() {
  %cst = arith.constant dense<0.000000e+00> : vector<1x1x1x8xf32>
  %c256 = arith.constant 256 : index
  %c32 = arith.constant 32 : index
  %c64 = arith.constant 64 : index
  %c1 = arith.constant 1 : index
  %c0 = arith.constant 0 : index
  %c1024 = arith.constant 1024 : index
  %c132096 = arith.constant 132096 : index
  %0 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c0) flags(ReadOnly) : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %0, 64 : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
  %1 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%c1024) flags(ReadOnly) : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %1, 64 : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>>
  %2 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%c132096) : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  memref.assume_alignment %2, 64 : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>>
  %workgroup_id_x = hal.interface.workgroup.id[0] : index
  %workgroup_count_x = hal.interface.workgroup.count[0] : index
  %workgroup_id_y = hal.interface.workgroup.id[1] : index
  %workgroup_count_y = hal.interface.workgroup.count[1] : index
  %3 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_id_x]
  %4 = affine.apply affine_map<()[s0] -> (s0 * 32)>()[%workgroup_count_x]
  scf.for %arg0 = %workgroup_id_y to %c1 step %workgroup_count_y {
    scf.for %arg1 = %3 to %c64 step %4 {
      %subview = memref.subview %2[%arg0, %arg1, 0, 0] [1, 32, 1, 8] [1, 1, 1, 1] : memref<1x64x1x8xf32, strided<[512, 8, 8, 1], offset: 33024>, #hal.descriptor_type<storage_buffer>> to memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_0 = memref.subview %1[%arg1, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<64x256x8x1xi8, strided<[2048, 8, 1, 1], offset: 1024>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      %subview_1 = memref.subview %subview_0[0, 0, 0, 0] [32, 256, 8, 1] [1, 1, 1, 1] : memref<32x256x8x1xi8, strided<[2048, 8, 1, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
      scf.for %arg2 = %c0 to %c32 step %c1 {
        %subview_2 = memref.subview %subview[0, %arg2, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x32x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
        %5 = scf.for %arg3 = %c0 to %c256 step %c1 iter_args(%arg4 = %cst) -> (vector<1x1x1x8xf32>) {
          %7 = vector.load %subview_1[%arg2, %arg3, %c0] : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
          %8 = arith.sitofp %7 : vector<8xi8> to vector<8xf32>
          %9 = vector.extract %arg4[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
          %10 = memref.load %0[%c0, %arg3, %c0, %c0] : memref<1x256x1x1xf32, #hal.descriptor_type<storage_buffer>>
          %11 = vector.broadcast %10 : f32 to vector<8xf32>
          %12 = vector.fma %8, %11, %9 : vector<8xf32>
          %13 = vector.broadcast %12 : vector<8xf32> to vector<1x1x1x8xf32>
          scf.yield %13 : vector<1x1x1x8xf32>
        }
        %6 = vector.extract %5[0, 0, 0] : vector<8xf32> from vector<1x1x1x8xf32>
        %subview_3 = memref.subview %subview_2[0, 0, 0, 0] [1, 1, 1, 8] [1, 1, 1, 1] : memref<1x1x1x8xf32, strided<[512, 8, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>> to memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>
        vector.store %6, %subview_3[%c0] : memref<8xf32, affine_map<(d0)[s0] -> (d0 + s0)>, #hal.descriptor_type<storage_buffer>>, vector<8xf32>
      }
    }
  }
  return
}

x86_64 assembly: https://gist.github.com/pzread/20f3e1e1f910d55016eadc33b9655853
arm_64 (pixel8) assembly: https://gist.github.com/pzread/aedded2e23e5b7cb6de05f7acfcf71d2

Looking forward to using https://github.com/llvm/llvm-project/pull/74817 : )

Yes, that is much much better! Let's wait for the unit dim patch to land. We can add it to vector lowering but we should start thinking about splitting that pass... it's getting huge and complex. Looking forward to have all of this landed! Thank you all!

Yes, that is much much better! Let's wait for the unit dim patch to land.

That's now available in LLVM, but now we need to wait for IREE's fork to be updated.

The LLVM changes have been picked up in IREE. This is ready for another review

It seems that there are couple changes after I reviewed it. I'll need to review it again before landing it, so I put a blocker for now.

Thanks, I'll request a review later. There are some regressions and I'm trying to figure out how to deal with them (maybe split up into different changes)

I did an investigation on the regression of Vit int8 model on Pixel. The heaviest and regressed (1.9s -> 2.38s in tracy) matmul dispatch is:
#config = #iree_codegen.lowering_config<tile_sizes = [[64, 64], [8, 16], [0, 0], [0, 0]]>
#config1 = #iree_codegen.lowering_config<tile_sizes = [[64, 64, 0], [64, 64, 0], [0, 0, 0], [8, 16, 0], [0, 0, 1], [0, 0, 0]]>
func.func @main_dispatch_7_matmul_197x768x768_i8xi8xi32() {
      %c0_i32 = arith.constant 0 : i32
      %0 = hal.interface.constant.load[0] : i32
      %1 = hal.interface.constant.load[1] : i32
      %2 = hal.interface.constant.load[2] : i32
      %3 = arith.index_castui %0 : i32 to index
      %4 = arith.index_castui %1 : i32 to index
      %5 = arith.index_castui %2 : i32 to index
      %6 = hal.interface.binding.subspan set(0) binding(0) type(storage_buffer) alignment(64) offset(%3) flags(ReadOnly) : !flow.dispatch.tensor<readonly:tensor<197x768xi8>>
      %7 = hal.interface.binding.subspan set(0) binding(1) type(storage_buffer) alignment(64) offset(%4) flags(ReadOnly) : !flow.dispatch.tensor<readonly:tensor<768x768xi8>>
      %8 = hal.interface.binding.subspan set(0) binding(2) type(storage_buffer) alignment(64) offset(%5) : !flow.dispatch.tensor<writeonly:tensor<197x768xi32>>
      %9 = flow.dispatch.tensor.load %6, offsets = [0, 0], sizes = [197, 768], strides = [1, 1] : !flow.dispatch.tensor<readonly:tensor<197x768xi8>> -> tensor<197x768xi8>
      %10 = flow.dispatch.tensor.load %7, offsets = [0, 0], sizes = [768, 768], strides = [1, 1] : !flow.dispatch.tensor<readonly:tensor<768x768xi8>> -> tensor<768x768xi8>
      %11 = tensor.empty() : tensor<197x768xi32>
      %12 = linalg.fill {lowering_config = #config} ins(%c0_i32 : i32) outs(%11 : tensor<197x768xi32>) -> tensor<197x768xi32>
      %13 = linalg.matmul {lowering_config = #config1} ins(%9, %10 : tensor<197x768xi8>, tensor<768x768xi8>) outs(%12 : tensor<197x768xi32>) -> tensor<197x768xi32>
      flow.dispatch.tensor.store %13, %8, offsets = [0, 0], sizes = [197, 768], strides = [1, 1] : tensor<197x768xi32> -> !flow.dispatch.tensor<writeonly:tensor<197x768xi32>>
      return
}

Before LLVMCPUVectorLowering, we got:
...
          %16 = vector.transfer_read %subview[%14, %arg3], %c0_i8 {in_bounds = [true, true]} : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8x1xi8>
          %17 = vector.transfer_read %subview_1[%arg3, %arg2], %c0_i8 {in_bounds = [true]} : memref<768x64xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xi8>
          %18 = arith.extsi %16 : vector<8x1xi8> to vector<8x1xi32>
          %19 = arith.extsi %17 : vector<16xi8> to vector<16xi32>
          %20 = vector.broadcast %19 : vector<16xi32> to vector<1x16xi32>
          %21 = vector.contract {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d2)>, affine_map<(d0, d1, d2) -> (d2, d1)>, affine_map<(d0, d1, d2) -> (d0, d1)>], iterator_types = ["parallel", "parallel", "reduction"], kind = #vector.kind<add>} %18, %20, %arg4 : vector<8x1xi32>, vector<1x16xi32> into vector<8x16xi32>
          scf.yield %21 : vector<8x16xi32>
...

As we are using the new pattern to remove unit dims on elementwise ops, during the process in LLVMCPUVectorLowering, we created a vector.shape_cast to turn vector<8x1xi32> into vector<8xi32>:
--- After lowering high level vector ops to lower level vector ops ---
...
          %16 = vector.transfer_read %subview[%14, %arg3], %c0_i8 {in_bounds = [true, true]} : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8x1xi8>
          %17 = vector.transfer_read %subview_1[%arg3, %arg2], %c0_i8 {in_bounds = [true]} : memref<768x64xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xi8>
          %18 = vector.shape_cast %16 : vector<8x1xi8> to vector<8xi8>
          %19 = arith.extsi %18 : vector<8xi8> to vector<8xi32>
          %20 = arith.extsi %17 : vector<16xi8> to vector<16xi32>
          %21 = vector.extract %19[0] : i32 from vector<8xi32>
          %22 = vector.broadcast %21 : i32 to vector<16xi32>
          %23 = vector.extract %arg4[0] : vector<16xi32> from vector<8x16xi32>
          %24 = arith.muli %22, %20 : vector<16xi32>
          %25 = arith.addi %24, %23 : vector<16xi32>
          %26 = vector.insert %25, %cst [0] : vector<16xi32> into vector<8x16xi32>
          %27 = vector.extract %19[1] : i32 from vector<8xi32>
          %28 = vector.broadcast %27 : i32 to vector<16xi32>
          %29 = vector.extract %arg4[1] : vector<16xi32> from vector<8x16xi32>
          %30 = arith.muli %28, %20 : vector<16xi32>
          %31 = arith.addi %30, %29 : vector<16xi32>
          %32 = vector.insert %31, %26 [1] : vector<16xi32> into vector<8x16xi32>
          %33 = vector.extract %19[2] : i32 from vector<8xi32>
          %34 = vector.broadcast %33 : i32 to vector<16xi32>
          %35 = vector.extract %arg4[2] : vector<16xi32> from vector<8x16xi32>
          %36 = arith.muli %34, %20 : vector<16xi32>
          %37 = arith.addi %36, %35 : vector<16xi32>
          %38 = vector.insert %37, %32 [2] : vector<16xi32> into vector<8x16xi32>
          %39 = vector.extract %19[3] : i32 from vector<8xi32>
          %40 = vector.broadcast %39 : i32 to vector<16xi32>
          %41 = vector.extract %arg4[3] : vector<16xi32> from vector<8x16xi32>
          %42 = arith.muli %40, %20 : vector<16xi32>
          %43 = arith.addi %42, %41 : vector<16xi32>
          %44 = vector.insert %43, %38 [3] : vector<16xi32> into vector<8x16xi32>
          %45 = vector.extract %19[4] : i32 from vector<8xi32>
          %46 = vector.broadcast %45 : i32 to vector<16xi32>
          %47 = vector.extract %arg4[4] : vector<16xi32> from vector<8x16xi32>
          %48 = arith.muli %46, %20 : vector<16xi32>
          %49 = arith.addi %48, %47 : vector<16xi32>
          %50 = vector.insert %49, %44 [4] : vector<16xi32> into vector<8x16xi32>
          %51 = vector.extract %19[5] : i32 from vector<8xi32>
          %52 = vector.broadcast %51 : i32 to vector<16xi32>
          %53 = vector.extract %arg4[5] : vector<16xi32> from vector<8x16xi32>
          %54 = arith.muli %52, %20 : vector<16xi32>
          %55 = arith.addi %54, %53 : vector<16xi32>
          %56 = vector.insert %55, %50 [5] : vector<16xi32> into vector<8x16xi32>
          %57 = vector.extract %19[6] : i32 from vector<8xi32>
          %58 = vector.broadcast %57 : i32 to vector<16xi32>
          %59 = vector.extract %arg4[6] : vector<16xi32> from vector<8x16xi32>
          %60 = arith.muli %58, %20 : vector<16xi32>
          %61 = arith.addi %60, %59 : vector<16xi32>
          %62 = vector.insert %61, %56 [6] : vector<16xi32> into vector<8x16xi32>
          %63 = vector.extract %19[7] : i32 from vector<8xi32>
          %64 = vector.broadcast %63 : i32 to vector<16xi32>
          %65 = vector.extract %arg4[7] : vector<16xi32> from vector<8x16xi32>
          %66 = arith.muli %64, %20 : vector<16xi32>
          %67 = arith.addi %66, %65 : vector<16xi32>
          %68 = vector.insert %67, %62 [7] : vector<16xi32> into vector<8x16xi32>
          scf.yield %68 : vector<8x16xi32>

This shape_cast won't be eliminated and eventually it is lowered into insert_strided_slice:
// -----// IR Dump After LLVMCPUVectorLowering (iree-llvmcpu-vector-lowering) //----- //
...
          // Lowered from vector.transfer_read
          %24 = memref.load %subview[%14, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %25 = vector.broadcast %24 : i8 to vector<1xi8>
          %26 = affine.apply affine_map<(d0) -> (d0 + 1)>(%14)
          %27 = memref.load %subview[%26, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %28 = vector.broadcast %27 : i8 to vector<1xi8>
          %29 = affine.apply affine_map<(d0) -> (d0 + 2)>(%14)
          %30 = memref.load %subview[%29, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %31 = vector.broadcast %30 : i8 to vector<1xi8>
          %32 = affine.apply affine_map<(d0) -> (d0 + 3)>(%14)
          %33 = memref.load %subview[%32, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %34 = vector.broadcast %33 : i8 to vector<1xi8>
          %35 = affine.apply affine_map<(d0) -> (d0 + 4)>(%14)
          %36 = memref.load %subview[%35, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %37 = vector.broadcast %36 : i8 to vector<1xi8>
          %38 = affine.apply affine_map<(d0) -> (d0 + 5)>(%14)
          %39 = memref.load %subview[%38, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %40 = vector.broadcast %39 : i8 to vector<1xi8>
          %41 = affine.apply affine_map<(d0) -> (d0 + 6)>(%14)
          %42 = memref.load %subview[%41, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %43 = vector.broadcast %42 : i8 to vector<1xi8>
          %44 = affine.apply affine_map<(d0) -> (d0 + 7)>(%14)
          %45 = memref.load %subview[%44, %arg3] : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>
          %46 = vector.broadcast %45 : i8 to vector<1xi8>
          %47 = vector.load %subview_2[%arg3, %arg2] : memref<768x64xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xi8>
          // Lowered from vector.shape_cast
          %48 = vector.insert_strided_slice %25, %cst {offsets = [0], strides = [1]} : vector<1xi8> into vector<8xi8>
          %49 = vector.insert_strided_slice %28, %48 {offsets = [1], strides = [1]} : vector<1xi8> into vector<8xi8>
          %50 = vector.insert_strided_slice %31, %49 {offsets = [2], strides = [1]} : vector<1xi8> into vector<8xi8>
          %51 = vector.insert_strided_slice %34, %50 {offsets = [3], strides = [1]} : vector<1xi8> into vector<8xi8>
          %52 = vector.insert_strided_slice %37, %51 {offsets = [4], strides = [1]} : vector<1xi8> into vector<8xi8>
          %53 = vector.insert_strided_slice %40, %52 {offsets = [5], strides = [1]} : vector<1xi8> into vector<8xi8>
          %54 = vector.insert_strided_slice %43, %53 {offsets = [6], strides = [1]} : vector<1xi8> into vector<8xi8>
          %55 = vector.insert_strided_slice %46, %54 {offsets = [7], strides = [1]} : vector<1xi8> into vector<8xi8>
          %56 = arith.extsi %55 : vector<8xi8> to vector<8xi32>
          %57 = arith.extsi %47 : vector<16xi8> to vector<16xi32>
          %58 = vector.extract %56[0] : i32 from vector<8xi32>
          %59 = vector.broadcast %58 : i32 to vector<16xi32>
          %60 = vector.extract %arg4[0] : vector<16xi32> from vector<8x16xi32>
          %61 = arith.muli %59, %57 : vector<16xi32>
          %62 = arith.addi %61, %60 : vector<16xi32>
          %63 = vector.insert %62, %cst_0 [0] : vector<16xi32> into vector<8x16xi32>
          %64 = vector.extract %56[1] : i32 from vector<8xi32>
          %65 = vector.broadcast %64 : i32 to vector<16xi32>
          %66 = vector.extract %arg4[1] : vector<16xi32> from vector<8x16xi32>
          %67 = arith.muli %65, %57 : vector<16xi32>
          %68 = arith.addi %67, %66 : vector<16xi32>
          %69 = vector.insert %68, %63 [1] : vector<16xi32> into vector<8x16xi32>
          %70 = vector.extract %56[2] : i32 from vector<8xi32>
          %71 = vector.broadcast %70 : i32 to vector<16xi32>
          %72 = vector.extract %arg4[2] : vector<16xi32> from vector<8x16xi32>
          %73 = arith.muli %71, %57 : vector<16xi32>
          %74 = arith.addi %73, %72 : vector<16xi32>
          %75 = vector.insert %74, %69 [2] : vector<16xi32> into vector<8x16xi32>
          %76 = vector.extract %56[3] : i32 from vector<8xi32>
          %77 = vector.broadcast %76 : i32 to vector<16xi32>
          %78 = vector.extract %arg4[3] : vector<16xi32> from vector<8x16xi32>
          %79 = arith.muli %77, %57 : vector<16xi32>
          %80 = arith.addi %79, %78 : vector<16xi32>
          %81 = vector.insert %80, %75 [3] : vector<16xi32> into vector<8x16xi32>
          %82 = vector.extract %56[4] : i32 from vector<8xi32>
          %83 = vector.broadcast %82 : i32 to vector<16xi32>
          %84 = vector.extract %arg4[4] : vector<16xi32> from vector<8x16xi32>
          %85 = arith.muli %83, %57 : vector<16xi32>
          %86 = arith.addi %85, %84 : vector<16xi32>
          %87 = vector.insert %86, %81 [4] : vector<16xi32> into vector<8x16xi32>
          %88 = vector.extract %56[5] : i32 from vector<8xi32>
          %89 = vector.broadcast %88 : i32 to vector<16xi32>
          %90 = vector.extract %arg4[5] : vector<16xi32> from vector<8x16xi32>
          %91 = arith.muli %89, %57 : vector<16xi32>
          %92 = arith.addi %91, %90 : vector<16xi32>
          %93 = vector.insert %92, %87 [5] : vector<16xi32> into vector<8x16xi32>
          %94 = vector.extract %56[6] : i32 from vector<8xi32>
          %95 = vector.broadcast %94 : i32 to vector<16xi32>
          %96 = vector.extract %arg4[6] : vector<16xi32> from vector<8x16xi32>
          %97 = arith.muli %95, %57 : vector<16xi32>
          %98 = arith.addi %97, %96 : vector<16xi32>
          %99 = vector.insert %98, %93 [6] : vector<16xi32> into vector<8x16xi32>
          %100 = vector.extract %56[7] : i32 from vector<8xi32>
          %101 = vector.broadcast %100 : i32 to vector<16xi32>
          %102 = vector.extract %arg4[7] : vector<16xi32> from vector<8x16xi32>
          %103 = arith.muli %101, %57 : vector<16xi32>
          %104 = arith.addi %103, %102 : vector<16xi32>
          %105 = vector.insert %104, %99 [7] : vector<16xi32> into vector<8x16xi32>
          scf.yield %105 : vector<8x16xi32>

One thing I tried is to simplify vector.insert_strided_slice into vector.extract + vector.insert_element as the source vector is just vector<1xi8>. In this way the vector.extract and fold with its source vector.broadcast : i8 -> vector<1xi8> and in the end we have a cleaner sequence of memref.load (from transfer read) : i8 -> vector.insert_element : i8 -> vector<8xi8>. However I found that there was no change in assembly output.
This is the dump of dispatch_7 with the changes in this PR:

IR dump: https://gist.github.com/pzread/febf53dc174583601fd05eba253f5d7c#file-main_dispatch_7_matmul_197x768x768_i8xi8xi32_after_dump-mlir
ARM64 assembly: https://gist.github.com/pzread/febf53dc174583601fd05eba253f5d7c#file-dispatch_7-s

@banach-space let me know if you spot some easy/clear issues here : ) Otherwise I might consider file a bug and ignore the regressions for now as we are seeing improvements in many other models.

This is the dump of dispatch_7:
* IR dump: https://gist.github.com/pzread/febf53dc174583601fd05eba253f5d7c#file-main_dispatch_7_matmul_197x768x768_i8xi8xi32_after_dump-mlir

* ARM64 assembly: https://gist.github.com/pzread/febf53dc174583601fd05eba253f5d7c#file-dispatch_7-s

@banach-space let me know if you spot some easy/clear issues here : )

This looks quite grim, but unrelated to the actual target, right? We should see similar poor code for other targets as well.
populateFlattenVectorTransferPatterns has some good patterns that could help getting rid of vector.shape_casts. But it might require some love - I am poking in that area, so can have a look later.

Otherwise I might consider file a bug and ignore the regressions for now as we are seeing improvements in many other models.

Sounds reasonable. Also, aren't these two Ops unrelated to the fact that DT is enabled?
              %29 = arith.extsi %26 : vector<8x1xi8> to vector<8x1xi32>
              %30 = arith.extsi %27 : vector<1x16xi8> to vector<1x16xi32>

Thanks for the feedback!

Otherwise I might consider file a bug and ignore the regressions for now as we are seeing improvements in many other models.

Sounds reasonable. Also, aren't these two Ops unrelated to the fact that DT is enabled?
              %29 = arith.extsi %26 : vector<8x1xi8> to vector<8x1xi32>
              %30 = arith.extsi %27 : vector<1x16xi8> to vector<1x16xi32>

Yes they are not related to DT enablement. The extra shape_cast happens because I'm currently using vector::populateDropUnitDimWithShapeCastPatterns(patterns); for all lowering paths:
In our previous DT-enabled case, that pattern provides improvement for this part:
%8 = vector.transfer_read %subview_2[%arg2, %arg3, %c0], %c0_i8 {in_bounds = [true]} : memref<32x256x8xi8, strided<[2048, 8, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8xi8>
%9 = vector.shape_cast %8 : vector<8xi8> to vector<8x1xi8>
%10 = arith.sitofp %9 : vector<8x1xi8> to vector<8x1xf32>

However for the DT-disabled case, it regresses this part:
%16 = vector.transfer_read %subview[%14, %arg3], %c0_i8 {in_bounds = [true, true]} : memref<?x768xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<8x1xi8>
%17 = vector.transfer_read %subview_1[%arg3, %arg2], %c0_i8 {in_bounds = [true]} : memref<768x64xi8, strided<[768, 1], offset: ?>, #hal.descriptor_type<storage_buffer>>, vector<16xi8>
%18 = arith.extsi %16 : vector<8x1xi8> to vector<8x1xi32>

The difference between two cases is that in the first case, transfer_read is reading a row-major vector<8xi8> and we want to get rid of shape_cast : vector<8xi8> -> vector<8x1xi8> and populateDropUnitDimWithShapeCastPatterns helps us by dropping unit dims on arith op and remove the shape_cast.
In the second case, transfer_read is reading a column-major vector<8x1xi8>, so even populateDropUnitDimWithShapeCastPatterns drops the unit dims on arith op, we still have a shape_cast to transform column-major vector<8x1xi8> into vector<8xi8>.
I think this is the situation where dropping unit dims doesn't always generate better code, depending on the context. I can try to apply populateDropUnitDimWithShapeCastPatterns only when DT is enabled. But in general I think we can make it smart enough to apply for all lowering paths.

Thanks for the explanation, that's incredibly useful!

In the second case, transfer_read is reading a column-major vector<8x1xi8>, so even populateDropUnitDimWithShapeCastPatterns drops the unit dims on arith op, we still have a shape_cast to transform column-major vector<8x1xi8> into vector<8xi8>.

Ah, so you reckon that the additional packing into vectors is the key contributor to perf degradation?
	ld1	{ v0.b }[1], [x8]
	ld1	{ v0.b }[2], [x8]
	ld1	{ v0.b }[3], [x8]
	ld1	{ v0.b }[4], [x6], #1
	ld1	{ v0.b }[5], [x8]
	ld1	{ v0.b }[6], [x11]
	ld1	{ v0.b }[7], [x17]

SVE enjoys sign-extending loads and I am hoping that those will unlock some perf improvements. But to leverage that we'll need to get rid of those vector.insert_strided_slices 🤔 . On a related note, this problem would go away if the reduction dimension for linalg.matmul was something other than 1. But that might require fine-tuning in other areas first (I've only had a brief look).

I did a little bit more profiling on Pixel 8 (but only with the general CPU features +dotprod as we are using in CI). The dispatch_7_matmul is 1.3X slower with this change.

Ah, so you reckon that the additional packing into vectors is the key contributor to perf degradation?

I don't have a conclusion yet but with/without packing into vectors results in the different multiplication instructions later.
If my understanding is correct, without packing into vectors (before this change), the code follows this pattern (instructions are re-ordered for readability):
// Load one element from LHS vector<8x1xi8>
ldrsb	w11, [x8]
// Broadcast into vector<4xi32>
dup	v4.4s, w11
// Elementwise multiplication with the extended RHS vector<16xi32> using mla
mla	v23.4s, v4.4s, v0.4s
mla	v7.4s, v4.4s, v1.4s
mla	v12.4s, v4.4s, v2.4s
mla	v17.4s, v4.4s, v3.4s
// Load the next element from LHS, broadcast, multiplication ... Repeated 8 times

With packing into vectors (after this change), the code becomes:
// Load 8 elements of LHS vector<8x1xi8> into vector<8xi8>
ldr	b0, [x8] (I guess this is the load of index 0?)
ld1	{ v0.b }[1], [x8]
ld1	{ v0.b }[2], [x8]
ld1	{ v0.b }[3], [x8]
ld1	{ v0.b }[4], [x6], #1
ld1	{ v0.b }[5], [x8]
ld1	{ v0.b }[6], [x11]
ld1	{ v0.b }[7], [x17]
// Elementwise multiplication with the extended RHS vector<16xi32> using smlal
smlal2	v8.4s, v2.8h, v0.h[0]
smlal	v5.4s, v2.4h, v0.h[0]
smlal2	v2.4s, v1.8h, v0.h[0]
smlal	v13.4s, v1.4h, v0.h[0]
...
smlal2	v4.4s, v2.8h, v0.h[7]
smlal2	v27.4s, v1.8h, v0.h[7]
smlal	v3.4s, v2.4h, v0.h[7]
smlal	v4.4s, v1.4h, v0.h[7]

So after packing LHS into a vector, we use smlal to do broadcast + multiplication instead of two separated steps.
I guess this is something we can file an issue and investigate later.

I think we no longer need this as the new UKernels have covered this case


training data

what are the wordvectors and charvectors
Do we need to use IOB2 corpus where both word and character embeddings are given
Can you please help through this?

I'm sorry for the delay in replying your comment.
Your description is perfectly right.

hi @s14t284,
Thanks for your great work.
How can I find the training dataset that reproduce your result?
Thanks

@tringn
I'm not the author of the paper that proposed this method.
So, I'm sorry if the following answer is wrong.
If you want to reproduce the results in the paper, buy the "mainichi news paper corpus", BCCWJ, and the corpus with extended named entity.
As far as I know, free Japanese NER corpuses don't exist.
However, we can make a pseudo Japanese NER corpus using KWDLC.
I made a script to make a Japanese NER corpus from KWDLC.
(https://github.com/s14t284/KWDLC2NERDataset)
Please use it if you like.


HCRC-6 | PoC use of RHHC

Description :sparkles:
This PR is meant to be a PoC example of how the RHHC library can be used to display headless-cms pages.

Implements component that are available from RHHC and deprecates them from this project
PoC that shows that the RHHC library is Server/Client compatible
PoC shows that the RHHC library is Next.js compatible

Codecov Report

Merging #231 (e640fc8) into master (48c62fa) will decrease coverage by 0.98%.
The diff coverage is 32.78%.

:exclamation: Current head e640fc8 differs from pull request most recent head bd7d202. Consider uploading reports for the commit bd7d202 to get more accurate results

@@            Coverage Diff             @@
##           master     #231      +/-   ##
==========================================
- Coverage   88.50%   87.51%   -0.99%
==========================================
  Files         120      117       -3
  Lines        3654     3605      -49
  Branches      804      834      +30
==========================================
- Hits         3234     3155      -79
- Misses        412      442      +30
  Partials        8        8

Impacted Files
Coverage Δ

src/common/components/meta/PageMeta.tsx
100.00% <ø> (ø)

src/headless-cms/components/CmsArticle.tsx
0.00% <0.00%> (ø)

src/headless-cms/components/utils.tsx
40.00% <40.00%> (ø)

...ess-cms/components/CmsPageSearch/CmsPageSearch.tsx
32.78% <50.00%> (ø)

src/domain/app/header/Header.tsx
88.09% <80.00%> (ø)

src/headless-cms/components/CmsPage.tsx
100.00% <100.00%> (ø)

src/common/components/htmlToReact/HtmlToReact.tsx
91.30% <0.00%> (-4.35%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 48c62fa...bd7d202. Read the comment docs.

This PR now includes the first implementation of the use of HCRC-library:

The translations are configured.
ArchivePage -component is used for subpage search.
Collections (with Grid-style) are implemented for Pages and Articles.
The CMS Pages are routed via cms-page -path and the CMS Articles are routed via cms-article -path.
The Page Sidebar is implemented.
Some server side rendering is taken care of with the HCRC.


Current and selected not working in mode "time"

Describe the bug
When the calendar is in mode "time", the current prop not working and I can't set default time to datepicker.
To Reproduce
// setTime is an useState

<DatePicker
    mode="time"
    options={{
            backgroundColor: '#f9f9f9',
            textHeaderColor: '#8ECA74',
            mainColor: '#8ECA74',
            selectedTextColor: '#fff'
          }}
     current={'2023-01-29 08:00'}
     selected={'2023-01-29 08:00'}
     onTimeChange={selectedTime => setTime(selectedTime)}
/>

Expected behavior
Default hour in 08 and minutes in 00.

I have same issue

same issue here


Support for array of objects / jitclasses

For my tasks I've implemented List jitclass based on np.array, but indeed I need array of Lists and even array of arrays of Lists:

List, e.g., List(4) - 4 is current length
AOL, e.g. [List(4), List(1), List(9)]
AOAOL, e.g.
[
[List(4), List(1), List(9)]
, [List(7), List(3)]
]

Straightforward implementation of AOL should be like
from numba.experimental import jitclass
from numba import njit, int32
import numba
import numpy as np

spec_List = [
    ('capacity', int32)
    ,('collection', int32[:])
    ,('p_top', int32)
]
@jitclass(spec_List)
class List:
    def __init__(self):
        pass

spec_ArrayOfLists = [
    ('capacity', int32)
    ,('l', List.class_type.instance_type[:])  # magic is here [:]
    ,('p_top', int32)
]

@jitclass(spec_ArrayOfLists)
class ArrayOfLists:
    def __init__(self):
        pass

aol = ArrayOfLists()

But I get an error
NotImplementedError: Failed in nopython mode pipeline (step: nopython mode backend)
Failed in nopython mode pipeline (step: nopython mode backend)
instance.jitclass.List#271415645c8<capacity:int32,collection:array(int32, 1d, A),p_top:int32> cannot be represented as a Numpy dtype

Every List length may vary, every array length may vary too but basically can be predicted in case of my task, so I can consider them as const. and there is workaround: we declare jitclass with 2-dim array where last dimension have size of maximal sub-List length (9 in example) and first dimension contains total count of lists (5 in the example), and there is special table with start indexes of every AOL: [0, 3] in the example. So my array should have shape of (5, 9). But of course it's better without workarounds :)

Thanks for the report. Just to check the intent... the issue here is that your code would need a NumPy ndarray with a dtype which is a typed.List? If so, I think this would end up in NumPy as an object array and Numba has no support for arrays of arbitrary objects.

typed.List is too slow for me (about 85 ns per operation, my own List - about 1-2 ns per op).

@ghostoid is the final content always a number (like int32 in the example above)? If so, jagged arrays provided by Awkward (https://github.com/scikit-hep/awkward-1.0) might be what you want.

For any further discussion, I'd suggest moving this over to Discourse: https://numba.discourse.group/


Can't play base64 audio

Be sure to search for your issue before opening a new one.
Current Behavior

Expected Behavior

Steps to Reproduce

Environment

URL attempting to play:
Browser:
Operating system:
jsFiddle example: https://jsfiddle.net/sv5x3ug1

Other Information

Would it be possible to provide an example? I've haven't heard of base64 audio before, can you provide some documentation or a specification? When it comes to audio, react-player uses the audio element so if base64 audio isn't supported by the element then there is nothing react-player can do to support the format/codec. Here is the list of supported codecs.


EP04-F04-US05 - Subject support content

As a user I would like to see which materials are recommended to the selected subject


Kraken: unknown minimum amount on XBT/XDG

I'm submitting a ...
[x] bug report
[ ] question about the decisions made in the repository
Action taken (what you did)
Started a live gekko on kraken XBT/XDG
Expected result (what you hoped would happen)
It trades.
Actual result (unexpected outcome)
Whenever gekko tries to buy/sell, it throws an error:
2018-09-16T17:21:57.092Z '[sticky order] FATAL ERROR' 'Order:Invalid price:Invalid price argument.'
2018-09-16T17:21:57.093Z Error: Order:Invalid price:Invalid price argument.
at rawRequest (/home/pi/gekko/exchange/node_modules/kraken-api/kraken.js:54:9)
at
at process._tickCallback (internal/process/next_tick.js:182:7)
Is there any way to get this to work?
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)
I'm running the latest version from the development branch.

I'm running the latest version from the development branch.

Could you specify this some more? I pushed some fixes here: 7d548fb these also include more logging. So if you were already running that please post more logs.

@askmike Ok, so I pulled the latest commit. The error is gone now. However, gekko's trades get rejected (?). The log looks like it tried to buy enough XDG but at a price of 0 XBT.
Kraken seems to be pretty ****** in ways of trading over API... 😢
The minimum order size for DOGE (XDG) is 3000 on kraken. This is mentioned here:
https://support.kraken.com/hc/en-us/articles/205893708-What-is-the-minimum-order-size-
018-09-17 04:18:17 (INFO):	Trader Received advice to go long. Buying  XDG
2018-09-17 04:18:17 (DEBUG):	Creating order to buy 6641.0396 XDG
2018-09-17 04:18:29 (DEBUG):	[ORDER] statusChange: SUBMITTED
2018-09-17 04:18:30 (DEBUG):	[ORDER] statusChange: OPEN
2018-09-17 04:18:38 (DEBUG):	[ORDER] statusChange: REJECTED
2018-09-17 04:18:38 (INFO):	[ORDER] summary: { price: 0,
  amount: 0,
  date: moment("1970-01-01T01:00:00.000"),
  side: 'buy',
  orders: 1 }
2018-09-17 04:18:38 (DEBUG):	syncing private data
2018-09-17 04:18:49 (WARN):	WARNING: exchange did not provide fee information, assuming no fees..
2018-09-17 04:26:32 (DEBUG):	syncing private data
2018-09-17 04:36:32 (DEBUG):	syncing private data
2018-09-17 04:46:32 (DEBUG):	syncing private data
2018-09-17 04:48:39 (INFO):	(PROFIT REPORT) original balance:		 0.00699057 XBT
2018-09-17 04:48:39 (INFO):	(PROFIT REPORT) current balance:		 0.00699057 XBT
2018-09-17 04:48:39 (INFO):	(PROFIT REPORT) profit:				 0.00000000 XBT (0.00000000%)
2018-09-17 04:56:32 (DEBUG):	syncing private data

Ah I replied to quickly, check my edit! So it might work, but not if the amount is less than the minimum. And we don't know what that is, I just pinged them about it: https://twitter.com/mikevanrossum/status/1041568645321678850

@H256 please try latest develop, I am still not 100% convinced since your error complains about price and what I fixed is about amount. Now I fixed the amount it should not really change the price calculations, but I'm not 100% sure about Kraken's error reporting here. Please open a new issue if the problem persists. (Since this one contains a lot of other things now).


notebook describing op.apply

We need a jupyter notebook explaining, perhaps with stupid examples, the mechanics of op.apply. in particular:

what it does:
** potentially triggers compilation
** potentially allocates data
**generation of default runtime values
** execution of code
accepted parameters (semantics of *_m, *_M, *_n, ...)
what else?

maybe this is related to PR #624 ?

this question appeared on Slack:

Just to check,  am I correct in thinking that when I run op_dev = Operator([stencil_dev]) followed by op_dev(time=timesteps, dt=dt), I am iterating my stencil by “timesteps” timesteps of size “dt”?

and here's my reply. We might reuse parts of it for the notebook we aim to write...
when you run an Operator, ideally you’d compute the timesteps in the compact interval [time_m, time_M], where m=min and M=max. time is a mere shortcut for time_M. Now, depending on the actual equations  being computed (and therefore depending on things such as the time order of your fields and what time derivatives have you taken), the actual timestep interval may be slightly smaller, to avoid out-of-bounds array accesses.
For example, if your equation is:
f[t+1, x] = f[t, x] + 1

and you run it with
Operator(...).apply(time=10)

then you’ll iterate over [0, 10], ie 11 iterations
if you instead run it with
Operator(...).apply(time_m=5, time_M=10)

then obviously it’ll be [5, 10]
However, consider now the following equation:
f[t+1, x] = f[t, x] + f[t-1, x]

Now the Operator cannot start at 0, as otherwise you’ll end up accessing f[-1, x]. So it will automatically set the iteration interval to [1, 10].
Note that this applies seamlessly to any dimension (not just time).
And this is supposed to work even if your Operator has 1000 complicated equations etc.
Also note this: https://github.com/opesci/devito/issues/520 . We haven’t described how this works properly :slightly_smiling_face: so apologies if the documentation is missing or is incomplete.
I think that print(Operator.apply.__doc__) may provide additional information


$textSplit() bug

Type

[x] Functions: $textSplit()

Code used:
$joinSplitText[ ]
$textSplit[1, 2, 3, 4;, ]

Info

Aoi.js version: git+https://github.com/aoijs/aoi.js#dev / 4.0.3
Node.js version: v12
Operating system: linux

ScreenShot

Description
Separator in $textSplit() is not working...

remove the space between , then try @falsisdev

Same

Same

@falsisdev try with a different seperator

But why is ,  not working?

ok ik
it will work in code but not in eval because
, is escaped to #COMMA#

hmmm i understand thx


CODEOWNERS

Checklist

[x] I have read the Contributor Guide
[x] I have read and agree to the Code of Conduct
[x] I have added a description of my changes and why I'd like them included in the section below

Description of Changes
Adding codeowner instructions to open sourcing guide.
Related Issues
https://github.com/EmbarkStudios/opensource/issues/2

Thanks!


Added more front-end tests

Added unit test for view function. Changed all the hard-coded routes to reverse_lazy. Added tests that check that registering a new user creates a new User in the database.

Coverage remained the same at 100.0% when pulling 950b05a3765824ea153be7e6f8f169bbf79db703 on front-end-1 into 08c906890e44c4a5dc04c560b9e37ef5f8e0cdeb on master.