[meta] Dota performance issues on Metal

Analyzing the Instruments profiles reveals quite a few things we could do better.
Native API calls

[x] Inefficient ObjectiveC selector lookups - #2145
[ ] Too many ObjectiveC messages, retains and releases. Roughly, our profile shows 5-7% dedicated to objc_msgSend versus 0.3-0.5% in MoltenVK. The difference in retain/release calls is of a similar scale.
[ ] Attribute access goes through messages - #2167

Metal backend issues

[ ] Too much heap (re)allocation. We should avoid any allocations at run-time by re-using the storage and/or using the iterators more aggressively.
[ ] Too many command buffers. The instrumented memory profile doesn't show command buffers being re-used. I suppose the queue just creates a new one every time until it reaches the capacity, and then starts trying to re-use the completed ones. This is undesired, and we may attempt to address it (temporarily) by playing with the capacity limit.
[ ] Dynamic creation of a render pass. We create one every time a render pass is started by copying one from the framebuffer and filling out all the clear values/operations. Apparently, this is super heavy, and we should probably avoid creating any Metal objects at normal run time.
[ ] Dynamic creation of depth-stencil states.
[ ] Binding descriptor sets is visibly slow. We should avoid doing work for repeated bindings, and need to explore ways to share heap-allocated data between sets of the same group.
[x] Non-existent semaphore frame synchronization  - #2143

Application concerns

The engine operates within an assumption that command buffer recording is cheap, while submission is expensive. They move the submission onto a separate thread, which doesn't appear to be saturated enough.

Things to investigate

[ ] Actual memory types/heaps used by an application. Maybe we could tweak the queries and/or ask Valve to fix those in order to use our exposed memory more efficiently. This is certainly a difference with Molten.

Other ideas:

consider caching function pointers to avoid selectors/objc_msgSend entirely (https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Articles/ocrtHowMessagingWorks.html#//apple_ref/doc/uid/TP40008048-CH104-SW1)
remove/cache/fix any cases where we are still performing Class::get etc. (i.e. the current autorelease pool bindings)

So getting counts of calls to objc_msgSend turned out to be more difficult than I thought. I started with lldb which was very, very slow. I switched to gdb, had to fix gdb and then it was still too slow. Finally I settled on using dtrace, which ran at a reasonable speed.
My results are:
MoltenVK: 47 thousand objc_msgSend/frame
gfx: 131 thousand objc_msgSend/frame

@jrmuizel wow, quite a big difference! How did you create this count with dtrace? Is there a way we could break it down by object name or selector to determine the difference?

Here's a per selector breakdown.
MoltenVK

  _addResource:                                                     0
  addResourceToList:                                                0
  allocatedSize                                                     0
  getCopyKernels                                                    0
  getUniqueID:clampMax:                                             0
  initWithDevice:options:args:argsSize:                             0
  initWithDevice:pointer:length:options:sysMemSize:vidMemSize:args:argsSize:deallocator:                0
  newBufferWithLength:options:                                      0
  sharedRef                                                         0
  copyFromBuffer:sourceOffset:toBuffer:destinationOffset:size:                0
  __new::                                                           0
  addPresentScheduledHandler:                                       0
  addScheduledHandler:                                              0
  containsObject:                                                   0
  didScheduledPresent                                               0
  error                                                             0
  immutablePlaceholder                                              0
  initWithArray:range:copyItems:                                    0
  initWithDrawablePrivate:layer:                                    0
  initWithObjects:count:                                            0
  member:                                                           0
  nextDrawable                                                      0
  present                                                           0
  presentDrawable:                                                  0
  presentScheduledInsertSeed                                        0
  priv                                                              0
  releasePrivateReferences:                                         0
  removeAllObjects                                                  0
  removeObjectsInArray:                                             0
  removeObjectsInRange:inArray:range:                               0
  setPresentScheduledInsertSeed:                                    0
  setWithArray:range:copyItems:                                     0
  vkpresent                                                         1
  blitCommandEncoder                                                1
  __new:::                                                          1
  initWithCapacity:                                                 1
  mutablePlaceholder                                                1
  setVisibilityResultMode:offset:                                   1
  HasDCL3                                                           1
  renderEncoderInitHWState                                          1
  contents                                                          2
  drawableSize                                                      2
  virtualAddress                                                    2
  copyFromBuffer:sourceOffset:sourceBytesPerRow:sourceBytesPerImage:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:                2
  pixelFormat                                                       2
  IABEmulationData                                                  2
  computeCommandEncoder                                             2
  dispatchThreadgroups:threadsPerThreadgroup:                       2
  getComputePipelineState                                           2
  setComputePipelineState:                                          2
  description                                                       2
  initWithFormat:locale:arguments:                                  2
  stringWithFormat:                                                 2
  _dispose                                                          3
  _xref_dispose                                                     3
  _submitAvailableCommandBuffers                                    3
  commandBufferDidComplete:startTime:completionTime:error:                3
  commit                                                            3
  commitCommandBuffer:wake:                                         3
  didCompleteWithStartTime:endTime:error:                           3
  didSchedule:error:                                                3
  didScheduleWithStartTime:endTime:error:                           3
  insertObjects:count:atIndex:                                      3
  releaseAllObjectsAndReset                                         3
  removeObject:                                                     3
  replaceObjectsInRange:withObjects:count:                          3
  replaceObjectsInRange:withObjectsFromArray:range:                 3
  setCommitted:                                                     3
  submitCommandBuffers:count:                                       3
  commandBufferWithUnretainedReferences                             3
  enqueue                                                           3
  enqueueCommandBuffer:                                             3
  getAndIncrementNumCommandBuffers                                  3
  initWithQueue:retainedReferences:                                 3
  initWithQueue:retainedReferences:synchronousDebugMode:                3
  initWithCommandBuffer:                                            4
  isCommitted                                                       4
  self                                                              4
  addCompletedHandler:                                              4
  countByEnumeratingWithState:objects:count:                        5
  setSamplerState:atIndex:                                          5
  _delayedEventMatchingMask:pull:                                   5
  _nextEventMatchingEventMask:untilDate:inMode:dequeue:                5
  distantFuture                                                     5
  drain                                                             5
  isEqualToDate:                                                    5
  isMainThread                                                      5
  nextEventMatchingMask:untilDate:inMode:dequeue:                   5
  timeIntervalSinceNow                                              5
  isNSArray__                                                       6
  setBuffer:offset:atIndex:                                         6
  addObject:                                                        6
  respondsToSelector:                                               6
  drawPrimitives:vertexStart:vertexCount:                           7
  isProfilingEnabled                                                7
  popDebugGroup                                                     7
  pushDebugGroup:                                                   7
  setFragmentBytes:length:atIndex:                                  7
  setStencilReferenceValue:                                         7
  getObjects:range:                                                 8
  _mutate                                                           9
  getUniqueID:                                                      9
  removeObjectAtIndex:                                             10
  distantPast                                                      11
  insertObject:atIndex:                                            12
  setTexture:atIndex:                                              14
  removeObjectsInRange:                                            14
  setVertexBytes:length:atIndex:                                   15
  setClearDepth:                                                   15
  setClearStencil:                                                 15
  timeIntervalSinceReferenceDate                                   17
  setClearColor:                                                   42
  depth                                                            70
  depthAttachment                                                  70
  height                                                           70
  stencilAttachment                                                70
  width                                                            70
  textureBarrier                                                   77
  setViewport:                                                     97
  count                                                           103
  setDepthBias:slopeScale:clamp:                                  107
  setHasDCL3:                                                     108
  initWithCommandBuffer:descriptor:stateDescriptor:               108
  renderCommandEncoderWithDescriptor:                             108
  renderPassDescriptor                                            108
  setVisibilityResultBuffer:                                      108
  commandBufferStorage                                            112
  fenceGroupPool                                                  112
  getScratchPageGPUAddress                                        112
  getStatLocations                                                112
  getStatOptions                                                  112
  globalTraceObjectID                                             112
  setStencilFrontReferenceValue:backReferenceValue:               114
  copyWithZone:                                                   116
  deviceRef                                                       116
  setLabel:                                                       116
  setBlendColorRed:green:blue:alpha:                              117
  colorAttachments                                                119
  objectAtIndexedSubscript:                                       119
  pTelemetry                                                      120
  resourceSize                                                    147
  sampleCount                                                     203
  objectAtIndex:                                                  209
  drawPrimitives:vertexStart:vertexCount:instanceCount:baseInstance:              217
  commandQueue                                                    221
  setCurrentCommandEncoder:                                       225
  device                                                          235
  autorelease                                                     236
  setScissorRect:                                                 238
  setDepthPlane:                                                  261
  setLevel:                                                       261
  setLoadAction:                                                  261
  setSlice:                                                       261
  setStoreAction:                                                 261
  setTexture:                                                     261
  texture                                                         262
  setDepthClipMode:                                               306
  setCullMode:                                                    306
  setFrontFacingWinding:                                          306
  setTriangleFillMode:                                            306
  setDepthStencilState:                                           314
  setRenderPipelineState:                                         314
  isStatEnabled                                                   346
  _descriptorPrivate                                              370
  class                                                           485
  init                                                            486
  allocWithZone:                                                  490
  akPrivateResourceList                                           543
  ioAccelResourceList                                             543
  drawIndexedPrimitives:indexCount:indexType:indexBuffer:indexBufferOffset:instanceCount:baseVertex:baseInstance:              594
  dealloc                                                         600
  alloc                                                           605
  endEncoding                                                     694
  setVertexSamplerState:atIndex:                                  763
  setVertexTexture:atIndex:                                       812
  fragmentIABEmulationData                                        820
  getRenderPipelineState                                          820
  vertexIABEmulationData                                          820
  setFragmentBuffer:offset:atIndex:                               971
  textureType                                                     990
  retain                                                         1059
  _descriptorAtIndex:                                            1086
  bindIndirectArgumentsWithFunctions:                            1419
  commandBufferResourceInfo                                      1512
  setFragmentSamplerState:atIndex:                               1722
  setVertexBuffer:offset:atIndex:                                2470
  release                                                        3026
  setFragmentTexture:atIndex:                                    3772
  objc_msgSend                                                  36863

Gfx:

  allocatePerUnitThreadCount                                        0
  compiler                                                          0
  extractPSCompilerOutput:                                          0
  extractVSCompilerOutput:                                          0
  fragmentFunction                                                  0
  fragmentFunctionKeyWithRenderPipelineDescriptor:options:previousStateVariant:fragmentKeySize:                0
  freeFragmentFunctionKey:fragmentKeySize:                          0
  freeVertexFunctionKey:vertexKeySize:                              0
  generate3DStateConstants:                                         0
  generate3DStateDS:                                                0
  generate3DStateHS:                                                0
  generate3DStatePS:                                                0
  generate3DStateSBE:                                               0
  generate3DStateTE:                                                0
  generate3DStateURB:                                               0
  generate3DStateVF:                                                0
  generate3DStateVS:                                                0
  generate3DStateWM:                                                0
  generateBindingWorkarounds:                                       0
  generateBlendState:                                               0
  generateMultisampleState:                                         0
  generateShaderState:                                              0
  initWithDevice:pipelineStateDescriptor:                           0
  initWithDevice:pipelineStateDescriptor:vertexVariant:fragmentVariant:                0
  inputInfoAndSize:                                                 0
  isPostTessellationVertexFunction:                                 0
  isTessellationFactorScaleEnabled                                  0
  label                                                             0
  newRenderPipelineStateWithDescriptor:error:                       0
  newRenderPipelineStateWithDescriptor:options:reflection:error:completionHandler:                0
  newRenderPipelineWithDescriptor:vertexVariant:fragmentVariant:                0
  newSerializedVertexDataWithFlags:error:                           0
  rasterSampleCount                                                 0
  serialQueue                                                       0
  serializeFragmentData                                             0
  setAlphaBlendOperation:                                           0
  setBlendingEnabled:                                               0
  setDestinationAlphaBlendFactor:                                   0
  setDestinationRGBBlendFactor:                                     0
  setFragmentFunction:                                              0
  setFragmentIABEmulationData:                                      0
  setInputPrimitiveTopology:                                        0
  setRgbBlendOperation:                                             0
  setSourceAlphaBlendFactor:                                        0
  setSourceRGBBlendFactor:                                          0
  setStepFunction:                                                  0
  setStride:                                                        0
  setVertexDescriptor:                                              0
  setVertexFunction:                                                0
  setVertexIABEmulationData:                                        0
  stepFunction                                                      0
  stepRate                                                          0
  stride                                                            0
  tessellationControlPointIndexType                                 0
  tessellationFactorFormat                                          0
  tessellationFactorStepFunction                                    0
  tessellationPartitionMode                                         0
  validateWithVertexFunction:error:                                 0
  vertexAttributes                                                  0
  vertexDescriptor                                                  0
  vertexFunction                                                    0
  vertexFunctionKeyWithRenderPipelineDescriptor:options:nextStageVariant:vertexKeySize:                0
  initNewTextureDataWithDevice:descriptor:sysMemSize:sysMemRowBytes:vidMemSize:vidMemRowBytes:args:                0
  initWithDevice:descriptor:sysMemSize:sysMemRowBytes:vidMemSize:vidMemRowBytes:args:argsSize:                0
  initWithDevice:shmemSize:                                         0
  initializeFastClearMCS:descPriv:texData:                          0
  maxTextureHeight2D                                                0
  maxTextureWidth2D                                                 0
  newTextureWithDescriptor:                                         0
  renderPassDescriptor                                              0
  setDepth:                                                         0
  setHeight:                                                        0
  setMipmapLevelCount:                                              0
  setRenderTargetArrayLength:                                       0
  setShmemSize:                                                     0
  setStorageMode:                                                   0
  setTextureType:                                                   0
  setUsage:                                                         0
  setWidth:                                                         0
  allocPushConstantSpace:                                           0
  debuggingEnabled                                                  0
  denormsEnabled                                                    0
  distributeRemainingURBAllocations:countAvailable:gtVariant:                0
  fastMathEnabled                                                   0
  framebufferReadEnabled                                            0
  functionConstantsDictionary                                       0
  generateSamplerToTextureMap:table:                                0
  getCString:maxLength:encoding:                                    0
  getVariantID                                                      0
  hash                                                              0
  initWithData:                                                     0
  initWithLibraryContainer:device:                                  0
  languageVersion                                                   0
  layouts                                                           0
  lengthOfBytesUsingEncoding:                                       0
  maxColorAttachments                                               0
  newFunctionWithName:                                              0
  newFunctionWithNameInternal:                                      0
  newLibraryWithSource:options:error:                               0
  objectForKey:                                                     0
  preprocessorMacros                                                0
  programObject                                                     0
  releaseCacheEntry:                                                0
  renderTargetArrayIndexType                                        0
  respondsToSelector:                                               0
  setLanguageVersion:                                               0
  setupURBState:gtVariant:                                          0
  supportsSampleCount:                                              0
  supportsTextureSampleCount:                                       0
  setPixelFormat:                                                   0
  initWithDevice:options:args:argsSize:                             0
  getCompilerOutputData                                             0
  requiresIABEmulation                                              0
  format                                                            0
  offset                                                            0
  validateWithDevice:                                               0
  _removeResource:                                                  0
  removeResourceFromList:                                           0
  functionConstants                                                 0
  initWithBytes:length:encoding:                                    0
  needsFunctionConstantValues                                       0
  limits                                                            0
  arrayLength                                                       0
  buffer                                                            0
  bufferBytesPerRow                                                 0
  bufferOffset                                                      0
  descriptorPrivate                                                 0
  initWithResource:                                                 0
  initWithTexture:pixelFormat:textureType:levels:slices:                0
  initWithTextureInternal:pixelFormat:textureType:levels:slices:compressedView:                0
  mipmapLevelCount                                                  0
  newTextureViewWithPixelFormat:textureType:levels:slices:                0
  numFaces                                                          0
  sharedRef                                                         0
  pixelFormat                                                       0
  generateClearRequirements                                         0
  generateSurfaceStateTarget                                        0
  getUniqueID:clampMax:                                             0
  initializeHWSurfaceState                                          0
  bufferIndex                                                       0
  __new::                                                           1
  containsObject:                                                   1
  didScheduledPresent                                               1
  immutablePlaceholder                                              1
  initWithArray:range:copyItems:                                    1
  initWithDrawablePrivate:layer:                                    1
  initWithObjects:count:                                            1
  isCapturing                                                       1
  member:                                                           1
  nextDrawable                                                      1
  present                                                           1
  presentScheduledInsertSeed                                        1
  priv                                                              1
  releasePrivateReferences:                                         1
  removeAllObjects                                                  1
  removeObjectsInArray:                                             1
  removeObjectsInRange:inArray:range:                               1
  setPresentScheduledInsertSeed:                                    1
  setWithArray:range:copyItems:                                     1
  sharedCaptureManager                                              1
  vkpresent                                                         1
  _addResource:                                                     1
  addResourceToList:                                                1
  attributeIndex                                                    1
  attributeType                                                     1
  isActive                                                          1
  setBufferIndex:                                                   1
  setFormat:                                                        1
  setOffset:                                                        1
  attributes                                                        1
  calculateNumURBEntries:stageIndex:countAvailable:                 1
  populateURBState:stageIndex:startAddress:gtVariant:                1
  blitCommandEncoder                                                1
  initWithCommandBuffer:                                            1
  patchType                                                         1
  allocatedSize                                                     1
  __new:::                                                          2
  addCompletedHandler:                                              2
  initWithCapacity:                                                 2
  mutablePlaceholder                                                2
  copyFromBuffer:sourceOffset:sourceBytesPerRow:sourceBytesPerImage:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:                2
  setVertexBytes:length:atIndex:                                    3
  initWithDevice:                                                   4
  self                                                              5
  countByEnumeratingWithState:objects:count:                        5
  _delayedEventMatchingMask:pull:                                   6
  _nextEventMatchingEventMask:untilDate:inMode:dequeue:                6
  distantFuture                                                     6
  drain                                                             6
  isEqualToDate:                                                    6
  isMainThread                                                      6
  nextEventMatchingMask:untilDate:inMode:dequeue:                   6
  timeIntervalSinceNow                                              6
  _dispose                                                          6
  _xref_dispose                                                     6
  insertObjects:count:atIndex:                                      7
  replaceObjectsInRange:withObjects:count:                          7
  replaceObjectsInRange:withObjectsFromArray:range:                 7
  submitCommandBuffers:count:                                       7
  _submitAvailableCommandBuffers                                    7
  isNSArray__                                                      10
  setClearStencil:                                                 10
  distantPast                                                      12
  getObjects:range:                                                15
  setClearDepth:                                                   15
  timeIntervalSinceReferenceDate                                   18
  setTexture:                                                      23
  HasDCL3                                                          39
  renderEncoderInitHWState                                         39
  setClearColor:                                                   52
  commit                                                           60
  commitCommandBuffer:wake:                                        60
  didSchedule:error:                                               60
  didScheduleWithStartTime:endTime:error:                          60
  enqueueCommandBuffer:                                            60
  setCommitted:                                                    60
  isCommitted                                                      60
  commandBufferDidComplete:startTime:completionTime:error:               61
  didCompleteWithStartTime:endTime:error:                          61
  removeObject:                                                    61
  addObject:                                                       62
  commandBufferWithUnretainedReferences                            65
  getAndIncrementNumCommandBuffers                                 65
  initWithQueue:retainedReferences:                                65
  initWithQueue:retainedReferences:synchronousDebugMode:               65
  releaseAllObjectsAndReset                                        65
  removeObjectAtIndex:                                             68
  insertObject:atIndex:                                            68
  depth                                                            68
  depthAttachment                                                  68
  height                                                           68
  stencilAttachment                                                68
  width                                                            68
  _mutate                                                          70
  removeObjectsInRange:                                            75
  isProfilingEnabled                                              120
  setHasDCL3:                                                     129
  initWithCommandBuffer:descriptor:stateDescriptor:               129
  renderCommandEncoderWithDescriptor:                             129
  copy                                                            130
  endEncoding                                                     131
  commandBufferStorage                                            131
  fenceGroupPool                                                  131
  getScratchPageGPUAddress                                        131
  getStatLocations                                                131
  getStatOptions                                                  131
  globalTraceObjectID                                             131
  resourceSize                                                    139
  colorAttachments                                                159
  objectAtIndexedSubscript:                                       163
  initWithDevice:depthStencilDescriptor:                          169
  initializeDepthStencilState:                                    169
  isDepthWriteEnabled                                             169
  newDepthStencilStateWithDescriptor:                             169
  setReadMask:                                                    188
  setWriteMask:                                                   188
  deviceRef                                                       196
  autorelease                                                     204
  sampleCount                                                     210
  setBlendColorRed:green:blue:alpha:                              212
  count                                                           240
  setViewport:                                                    250
  commandQueue                                                    260
  pTelemetry                                                      261
  setCurrentCommandEncoder:                                       262
  setLoadAction:                                                  273
  setStoreAction:                                                 273
  texture                                                         274
  drawPrimitives:vertexStart:vertexCount:instanceCount:baseInstance:              298
  new                                                             324
  backFaceStencil                                                 339
  frontFaceStencil                                                339
  readMask                                                        339
  getUniqueID:                                                    369
  depthCompareFunction                                            383
  copyWithZone:                                                   393
  _descriptorPrivate                                              404
  setDepthClipMode:                                               407
  setRenderPipelineState:                                         410
  setStencilFrontReferenceValue:backReferenceValue:               438
  stencilCompareFunction                                          460
  device                                                          465
  setScissorRect:                                                 486
  depthFailureOperation                                           522
  stencilFailureOperation                                         522
  isStatEnabled                                                   523
  setDepthBias:slopeScale:clamp:                                  550
  setDepthStencilState:                                           597
  depthStencilPassOperation                                       602
  akPrivateResourceList                                           649
  ioAccelResourceList                                             649
  length                                                          679
  writeMask                                                       679
  drawIndexedPrimitives:indexCount:indexType:indexBuffer:indexBufferOffset:instanceCount:              712
  alloc                                                           865
  init                                                            873
  allocWithZone:                                                  875
  dealloc                                                         912
  class                                                           925
  fragmentIABEmulationData                                       1011
  getRenderPipelineState                                         1011
  vertexIABEmulationData                                         1011
  textureType                                                    1243
  isEqual:                                                       1288
  _descriptorAtIndex:                                            1300
  objectAtIndex:                                                 1670
  bindIndirectArgumentsWithFunctions:                            1779
  commandBufferResourceInfo                                      1807
  setFragmentBuffer:offset:atIndex:                              1964
  setFragmentSamplerState:atIndex:                               2874
  setVertexSamplerState:atIndex:                                 2874
  setVertexBuffer:offset:atIndex:                                3843
  setFragmentTexture:atIndex:                                    5615
  setVertexTexture:atIndex:                                      5615
  retain                                                        49918
  release                                                       51151
  objc_msgSend                                                 161044

And the dtrace script to produce it
#pragma D option quiet
BEGIN {
	self->count = 0;
}

pid$target::objc_msgSend:entry
{
	@msgCount[copyinstr(arg1)] = count();
	@msgCount["objc_msgSend"] = count();
}

pid$target::__ZN15portability_gfx5impls18gfxQueuePresentKHR17h17b127701ff0fa81E:entry
{
        self->count++;
	@msgCount["vkpresent"] = count();
        normalize(@msgCount, self->count);

}

I took a look at where the majority of the -release calls were and here's the breakdown of the top callers per frame:
  MTLIGSamplTexture                                                56
  MTLIGAccelCommandQueue                                           59
  MTLIGAccelBuffere                                                69
  MTLIGAccelTextur                                                 70
  MTLIGAcceler                                                     87
  MTLRenderPassColorAttachmentDescriptorArrayInternal               89
  MTLRenderPassColorAttachmentDescriptorInternal                   94
  MTLIGAccelRenderCommandEncoder                                  113
  MTLRenderPassDescriptorInternal                                 203
  MTLIOAccelPooledResource                                        227
  NSAutoreleasePool                                               294
  MTLIGAccelCommandBuffer                                         298
  MTLIGAccelDevice                                                303
  MTLIGDepthStencilState                                         1087
  MTLIGRenderPipelineState                                       1103
  MTLIGSampler                                                  10834
  MTLIGAccelBuffer                                              13325
  MTLIGAccelTexture                                             21279

  total                                                         50958

Calling -release on MTLIGAccelTexture 21 thousand times per frame seems too high.

For comparison here's what it looks like with MoltenVK
  MTLIOAccelPooledResource                                         11
  MTLRenderPassStencilAttachmentDescriptorInternal                 75
  MTLRenderPassDepthAttachmentDescriptorInternal                   75
  MTLRenderPassColorAttachmentDescriptorArrayInternal             141
  MTLRenderPassDescriptorInternal                                 142
  __NSCFConstantString                                            147
  MTLRenderPassColorAttachmentDescriptorInternal                  153
  MTLIGAccelCommandBuffer                                         154
  MTLIGAccelDevice                                                292
  MTLIGAccelTexture                                               304
  MTLIGRenderPipelineState                                        445

  total                                                          1972

I did something similar last night (just logging all objc_msgSend calls):

Thanks @jrmuizel! I think we'll try to avoid these clones entirely by working with the pointer of the native object directly (through their id) and upgrading them (probably transmute) to *Ref types whenever we need to do something with them. We shouldn't need this reference counting on the Objective-C side at all for these objects.

We should also consider disabling all interactions with the capture manager in release, or hide behind a feature.

Dota also attempts to use VK_FORMAT_BC1_RGB_UNORM and VkPipelineCache, so we should ensure we support them.

Too many command buffers

I tried to reduce the capacity and didn't see any visible impact. Created #2180 to possibly address this properly in the future.

With all the optimization coming (e.g. #2183) the heaviest call stack that Instruments show is about creating a render pass. We do this on Vulkan render passes of course, but also quite a few times for the VkCmdClearImage - once for each (slice, level) pair. There is a faster path (see CLEAR_IMAGE_ARRAY) currently blocked by OSX driver issues (partly filed as 40769756). What I do wonder about though is how our render pass creation count is different from MoltenVK at the moment. The target message is [MTLDebugCommandBuffer remderCommandEncoderWithDescriptor].

Just updated a bunch of items in the issue body.

What I do wonder about though is how our render pass creation count is different from MoltenVK at the moment.

Those counts are already provided by @jrmuizel . We call it 129 times versus 108 in Molten. Still a difference to investigate, but not one to be too worried about.

The engine operates within an assumption that command buffer recording is cheap, while submission is expensive.

I clarified the threading model now, and there appears to be no good reason for us to be CPU bound, on a main thread where we only run 1/3 of time... Dota just needs to use more threads for the job execution.

Too much heap (re)allocation

Some of it is addressed in #2185. Iterators are fine where they are applicable. In other places we have to resort to SmallVec, which is never optimal and often causes too much data moves... Also we really need dynamic on-stack allocation (alloca) to help here - https://github.com/rust-lang/rust/issues/48055.

Too many ObjectiveC messages

With the actual threading model clarified, I now see ~5% time spent by MoltenVK in objc_msgSend. Thus, I don't think there is any difference in how effective we are at message sending, but rather in a sheer number of the sends we do.

Here is what our frame looks like when in immediate recording mode:

Some observations:

a frame consists of one large submission (fenced), then present call, then one dummy submission (also fenced)
all the work is done in that large submission, but it's actually spread over 2 different frames if we believe the colors. So a frame submission has some command buffers from the next frame?..
the driver doesn't bother doing any work until we actually commit() - #2232
a lot of command buffers sit idle after being recorded
there isn't that much overlap between GPU and CPU work happening, unfortunately, so there is a big room for optimization, presumably on the application side

Ran Vulkan Dota 2 on Windows using RGP (trace file) to compare:

I believe this can be closed now, our results were published on the blog


Investigating if VM cleaner is broken

I found some machines (around 200) on the ubuntu.2204.amd64.open.rt queue that were not hear beating, the oldest was from 09/22 and that wasn't removed by the vm cleaner, which makes me wonder if is not working and if this is impacting more queues, investigating

Release Note Category

[ ] Feature changes/additions
[ ] Bug fixes
[ ] Internal Infrastructure Improvements

Release Note Description

Update:
After looking at AI the VM cleaner is running, is not running every hour, as it should, is not even running once per day Go to Log Analytics and run query.
this is very problematic; the reason is that the way the VM cleaner works is that:

it runs one time and store the name of new machines not heart beating
if there are machines that haven't been heart beating during two checks it deletes it

this works when it runs every hour because at most a useless machine is running for 2hours or so, but as things are right now a machine can be around for days, maybe 3 or 4 days.
I will keep investigating.

Update:
we are deleting every machine individually and waiting for it to get deleted.
I am investigating how can we approach differently:

If there is a wait to delete the machines and not waiting for them to be deleted
At least send all the delete request in one batch

PR: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/34286


Shebang support for Mir files?

Would you consider to support shebangs? I think that this will make Mir files feel much more "native" and easier to use.
From what I understand, there should be support to both add a shebang when you generate the final "bmir" file and then, there should be support to "skip" the shebang in the beginning of the file when it is given for interpretation (jit or not).
Let me know what you think!

Sorry for delay with the answer.  Usually I am very busy at the end of year.
I never thought about this.  I guess it is not hard to implement.  So I am not against this.  I has to much work on my plate to implement it myself.  But I am ready to consider the corresponding PR.
Imho shebangs is more oriented to textual files.  As MIR-textual file has # as a comment start, shebangs should already work for MIR-textual files.  But binary files are much smaller.

Hello! Glad you are doing fine my friend, I haven't seen you in a while!
Well, shebangs can be used with Mir so we can directly execute them through shell or through the "execve" system call. It will make Mir files feel more "native" in my opinion, and it will make it seem more professional overall.
I suppose it would be easy to implement. Unless I see something, there should be 2 steps.

The final Mir executable file should contain the shebang.
When the Mir loader (is this how is it called?) is called on a file, it should see if the first character is '#' and then, skip until the end of the line (that would be the separator). Then, it normally executes.

I would be glad to help if you would tell me which places I should look at and change, as I never looked into Mir's code before, and I'm not exactly the best at looking at source code of other people.  So in the positive side, it would be a nice first experience! ;)
I should also mention that I want to use Mir alongside LLVM for Nemesis, so the time has come where I don't tell you how I plan to use Mir, but I actually go and use it! I will start building the frontend soon so we're going to see some real world usage!

Hello there,
I think another option could be the binfmt_misc , but we still need a mir executable/loader, which shouldn't be too hard to do

Hello there,
I think another option could be the binfmt_misc , but we still need a mir executable/loader, which shouldn't be too hard to do

Hello and happy new year!!!
This looks very interesting! I have seen a video about RISC-V assembly on Linux and at some point, he is able to run the problem without invoking "qemu-riscv64" before the name of the program. I suppose, this is what it's used there!
Compared to shebangs, this has the advantage of been more clean and not requiring extra work to happen. The disadvantage however is that it's not portable and every system will have to add it. It is a small disadvantage however if the loader is packed for the package managers and automatically add it every time it is installed (same way that happens on Arch with the "qemu-user-binfmt" package for QEMU)!
As for the loader, c2m already has the loader built-in, so I suppose that there will need to happen a small change to "separate" the loader into its own project. I may be wrong however and require more work, I don't know the codebase and structure of Mir. It can even be tested NOW using c2m -eg as the loader!

I added some modifications to the c2mir driver, with binfmt_misc line :mir:M::MIR::/usr/local/bin/c2m-el:P I am able to run:
$ cat test.c
#include <stdio.h>

int main(int argc, char **argv)
{
  for(int i=0; i<argc; i++)
    puts(argv[i]);

  return 0;
}
$ c2m test.c -o test.bmir
$ chmod +x test.bmir
$ ./test.bmir hello world
Code generation of function main:
  Code generation for main: 62 MIR insns (addr=7fc811ce20e0, len=256) -- time 0.19 ms
./test.bmir
hello
world
$ cat /proc/sys/fs/binfmt_misc/mir
enabled
interpreter /usr/local/bin/c2m-el
flags: P
offset 0
magic 4d4952

I added some modifications to the c2mir driver, with binfmt_misc line :mir:M::MIR::/usr/local/bin/c2m-el:P I am able to run:
...
enabled
interpreter /usr/local/bin/c2m-el
flags: P
offset 0
magic 4d4952

Damn, you guys make me feel jealous!  Now, all that remains is for Vladimir to "separate" the loader and make it its own executable. This would also allow "c2m" to be compiled with Mir (which now that I think about it, you can compile it, but you won't be able to use it afterwards as the loader needs to be an ELF file and now the loader is part of "c2m").
One thing to think about that now is the fact that the "binfmt_misc" file will hardcode the opinion passed to the loader. Now, we can say that in a real scenario (release binary), we can forget the interpretation mode. However, there are still two JIT modes one been lazy compilation (-el, the one we used) and the second been full compilation (-eg) which compiles everything before it is called (or something like that? I don't know exactly how it works, tbh). I like the idea of the lazy mode, but I wonder if noticeable slowbacks will be noticed when it is used compared to the other mode.

Don't feel bad xD we all go through "phases" of some sort,
Anyway, this was mostly a PoC, I used the c2mir sub project because it already had the symbol resolver and the functions to make it compile (top API doesn't seem to have that?)
Regarding the hardcoded option, from the expected command line usage (some-command arg1 ...) we can't really pass the -ei/-eg/-el flag to MIR without "colliding" with the actual command arguments.
Although I'd suggest a environment variable to configure which execution to run (interp/JIT/lazy), that the MIR loader would see:
MIR_EXE_WAY=interp some-command arg1 ...
# or
export MIR_EXE_WAY=lazy
# ... Later on ...
some-command arg1...

(Here MIR_EXE_WAY is an example name, I'm taking suggestions to replacements XD and some-command is a MIR binary)
In general you'd have to modify the binary, even with shebang, to specify the execution "way" to go for (I believe).
A new loader would be advantageous here indeed, it would also remove the need for the .bmir file extension,
I may be able to look into it next week

Don't feel bad xD we all go through "phases" of some sort,

Thank you! I'm slowly getting over it ;)

Regarding the hardcoded option, from the expected command line usage (some-command arg1 ...) we can't really pass the -ei/-eg/-el flag to MIR without "colliding" with the actual command arguments.
Although I'd suggest a environment variable to configure which execution to run (interp/JIT/lazy), that the MIR loader would see:
MIR_EXE_WAY=interp some-command arg1 ...
# or
export MIR_EXE_WAY=lazy
# ... Later on ...
some-command arg1...

That's a great idea! We could have a default mode (I suggest "lazy") when the environment variable is not set, and then have the environment variable be set or modified (if already set) when you want to use a different mode.
The only thing I would change for that is the "JIT" mode to be used with lowercase letter ("jit"). "interp" could also be changed to the shorter "exec" which also makes sense as this is practically what the interpreter does.

(Here MIR_EXE_WAY is an example name, I'm taking suggestions to replacements XD and some-command is a MIR binary)

Another name I thought about is: MIR_TYPE to make it shorter and keeps making sense.

In general you'd have to modify the binary, even with shebang, to specify the execution "way" to go for (I believe).

And for that reason, binfmt_misc is better, as it's easier to modify it, and we don't have to write more code!

A new loader would be advantageous here indeed, it would also remove the need for the .bmir file extension,

Yeah, removing the extension would also make it feel better and more like a "system" file!

I may be able to look into it next week

Sounds awesome! Glad you are having fun with the project! :)

Hello,
I created the mir-run binary that is dedicated to running MIR binaries from the binfmt_misc interface.
Here is an example of running it:
[user@optiplex /tmp]$ cat t.c
#include <stdio.h>

int main(int argc, char**argv)
{
        for(int i=0;i<argc;i++)
                puts(argv[i]);
        return 0;
}
[user@optiplex /tmp]$ c2m t.c -o test
[user@optiplex /tmp]$ chmod +x test
[user@optiplex /tmp]$ sudo su
[root@optiplex tmp]# line=:mir:M::MIR::/usr/local/bin/mir-run:P
[root@optiplex tmp]# echo $line > /proc/sys/fs/binfmt_misc/register
[root@optiplex tmp]# cat /proc/sys/fs/binfmt_misc/mir
enabled
interpreter /usr/local/bin/mir-run
flags: P
offset 0
magic 4d4952
[root@optiplex tmp]#
exit
[user@optiplex /tmp]$ ./test
./test
[user@optiplex /tmp]$ ./test hello world
./test
hello
world
[user@optiplex /tmp]$ MIR_TYPE=jit ./test hello world
./test
hello
world
[user@optiplex /tmp]$ MIR_TYPE=lazy ./test hello world
./test
hello
world
[user@optiplex /tmp]$ MIR_TYPE=not-valid ./test hello world
warning: unknown MIR_TYPE 'not-valid', using default one
./test
hello
world

Here is an example with shared libraries:
[user@optiplex /tmp/mir-with-libs]$ bat lib.h lib.c test.c
───────┬────────────────────────────────────────────────────────────────────────────────────────────
       │ File: lib.h
───────┼────────────────────────────────────────────────────────────────────────────────────────────
   1   │
   2   │ #ifndef _TEST_LIB_H_
   3   │ #define _TEST_LIB_H_
   4   │
   5   │ extern int print_integer(int integer);
   6   │
   7   │ #endif
───────┴────────────────────────────────────────────────────────────────────────────────────────────
───────┬────────────────────────────────────────────────────────────────────────────────────────────
       │ File: lib.c
───────┼────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #include <stdio.h>
   2   │
   3   │ #include "lib.h"
   4   │
   5   │ int print_integer(int integer) {
   6   │     return printf("%d\n", integer);
   7   │ }
───────┴────────────────────────────────────────────────────────────────────────────────────────────
───────┬────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test.c
───────┼────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #include "lib.h"
   2   │
   3   │ int main() {
   4   │     print_integer(666);
   5   │     print_integer(42);
   6   │     print_integer(-5);
   7   │     return 0;
   8   │ }
───────┴────────────────────────────────────────────────────────────────────────────────────────────
[user@optiplex /tmp/mir-with-libs]$ gcc -c -fPIC lib.c
[user@optiplex /tmp/mir-with-libs]$ gcc -shared lib.o -o liblib.so
[user@optiplex /tmp/mir-with-libs]$ c2m test.c -o test -L. -llib
[user@optiplex /tmp/mir-with-libs]$ chmod +x test
[user@optiplex /tmp/mir-with-libs]$ MIR_LIB_DIRS=$(pwd) MIR_LIBS=lib ./test
666
42
-5
[user@optiplex /tmp/mir-with-libs]$

That's so awesome!!!! Your work is amazing! I'll leave this issue open until Vladimir merges your pull request!
Now, we need system call support and been able to create libraries and MIR is set for a general purpose backend!!! You guys are awesome!!! You give my motivation to work on Nemesis and hopefully have it ready at some point!

Regarding libraries, or modules (which could probably be another issue), and here I'm assuming these libraries will also be output in MIR,
There are a couple of ways that could be implemented (from the top of my head), depending as well on how you decide to implement them in Nemesis:
One way would be to have a custom C function that takes the name of the library (lib1.bmir), reads the file (searches somewhere predefined) into the MIR context and links it. The function can be directly implemented in the loader (a variant of the mir-run.c) or pre-loaded from a library (using the MIR_LIBS env var). The import_resolver should be able to find it (if in MIR_LIBS) or an extra else if in case of mir-run modification.
Still a bit similar to that one

Regarding libraries, or modules (which could probably be another issue), and here I'm assuming these libraries will also be output in MIR,
There are a couple of ways that could be implemented (from the top of my head), depending as well on how you decide to implement them in Nemesis:
One way would be to have a custom C function that takes the name of the library (lib1.bmir), reads the file (searches somewhere predefined) into the MIR context and links it. The function can be directly implemented in the loader (a variant of the mir-run.c) or pre-loaded from a library (using the MIR_LIBS env var). The import_resolver should be able to find it (if in MIR_LIBS) or an extra else if in case of mir-run modification.
I guess this could be the easier option as you can create a wrapper script for binfmt_misc that sets the environment variables and then calls mir-run.
Then you have other options such as:
* an array of (BMIR) libraries to load, set as a module-level data item (array) that the custom loader shall read and link to MIR context;

* if the function/variable you're accessing from has the format `<package>.<package>.<library>.<funcname>`, a custom `import_resolver` could find the BMIR file from the prefix, read & link (a bit trickier I guess)

I hope that's useful, best of luck

Thank you so much for the information. However, if I am to use libraries, I'll wait until and if Mir gives native support for them (think how ELF has one). Until then, I'll keep working in the frontend and wait!


Prevents approx error when using custom eval

Prevents xgboost in R from erroring when the following conditions are met:

You are using a custom evaluation function
The score is not reproducible, most of the times at 1e-13 precision
You are using the default early_stopping_rounds callback (thus, early_stopping_rounds parameter is setup)

Allowing leniency in the best score comparison (by 1e-12) allows to prevent xgboost from stopping and cancelling the model right when training the model is over, even if R reports the difference between best_score and attr_best_score is virtually 0 when printed (but will ever say they are not equal). Hence, the user do not lose his/her model at the end (especially useful if the model takes hours/days to train or if it is run in a batch).

Thanks @Laurae2 . That explains #1564 .
However, it highlights a deeper numerical issue: the xgboost attribute-setting code uses as.character, which admits that "conversion to character and back to numeric may change the number" (see details of https://stat.ethz.ch/R-manual/R-devel/library/base/html/character.html ). A suggested workaround is to store numerical attributes using format(x, digits = 16). The python side might need a similar attribute precision increase.

@khotilov @Laurae2 I think discussion in https://github.com/dmlc/xgboost/pull/1564 summarizes this issue and we can close this PR now. I think drat repo has been updated with @khotilov's bugfix


[BUG] Error running upgrade on a 0.3 cluster: missing shared-config

Describe the bug
When trying to upgrade a 0.3 cluster to develop, the user should receive a message that this type of upgrade is not supported (according to PR https://github.com/epiphany-platform/epiphany/pull/1476). In fact epiphany throws an error right after running the command epicli upgrade because of missing shared-config in the manifest file.
To Reproduce
Steps to reproduce the behavior:

Deploy a 0.3 cluster using image epiphanyplatform/epicli:0.3.1
Execute epicli upgrade -b /path/to/build/dir from develop branch

Expected behavior
The upgrade aborts with the message Your Kubernetes version ({{ cluster_version }}) is not supported by this version of Epiphany which requires at least version 1.14.6 (Epiphany v0.4.4). For more information, refer to the documentation. because the assertion was not satisfied.
Actual behavior
Epicli fails with ERROR epicli - Expected one element but received: 0
OS (please complete the following information):

OS: [all]

Cloud Environment (please complete the following information):

Cloud Provider [all]

Additional context
Debug log:
13:16:28 ERROR epicli - Expected one element but received: 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cli/epicli.py", line 127, in main
    return args.func(args)
  File "/usr/local/lib/python3.7/site-packages/cli/epicli.py", line 227, in run_upgrade
    return engine.upgrade()
  File "/usr/local/lib/python3.7/site-packages/cli/engine/UpgradeEngine.py", line 54, in upgrade
    ansible_runner.upgrade()
  File "/usr/local/lib/python3.7/site-packages/cli/engine/ansible/AnsibleRunner.py", line 137, in upgrade
    ansible_vars_generator.generate()
  File "/usr/local/lib/python3.7/site-packages/cli/engine/ansible/AnsibleVarsGenerator.py", line 74, in generate
    self.populate_group_vars(ansible_dir)
  File "/usr/local/lib/python3.7/site-packages/cli/engine/ansible/AnsibleVarsGenerator.py", line 99, in populate_group_vars
    shared_config_doc = self.get_shared_config_from_manifest()
  File "/usr/local/lib/python3.7/site-packages/cli/engine/ansible/AnsibleVarsGenerator.py", line 149, in get_shared_config_from_manifest
    shared_config_doc = select_single(manifest_docs, lambda x: x.kind == 'configuration/shared-config')
  File "/usr/local/lib/python3.7/site-packages/cli/helpers/doc_list_helpers.py", line 31, in select_single
    raise ExpectedSingleResultException("Expected one element but received: " + str(elements_count))
cli.helpers.doc_list_helpers.ExpectedSingleResultException: Expected one element but received: 0

@toszo can you get information if we know about any 0.3 cluster existing out there in the wild?

@przemyslavic there are such clusters.

The fix has been tested. The error described above no longer occurs.


build: Refactor Taskfile.yml

Refactor Taskfile.yml to provide these commands that build and serve their product:

run website
run playground
run book

The new install-playground-dependencies-all task forces the full "npm install..." with all its options All the other tasks rely on install-playground-dependencies that checks if packages.json has been modified and re-runs the full "npm install..." as needed
Breaking change: tasks run-web and run-playground-cached are no longer needed and commented out

I think this is overall very good, but I think it has one distinct large disadvantage: that changing the rust code and running task run-playground will not cause a rebuild. Is that accurate? That seems quite important.

The comment & footnote in https://github.com/PRQL/prql/pull/2907#issuecomment-1698270695 references an approach using webpack directly[^1], which I think would provide a way to let JS know when we need to rebuild files, which I think would allow for only rebuilding when necessary — both the JS and the rust files.
(But I don't know this well, I'm very green in JS)

... changing the rust code and running task run-playground will not cause a rebuild.

That would be bad. Would you suggest a small change I could make to a rust file that should cause a rebuild?

That would be bad. Would you suggest a small change I could make to a rust file that should cause a rebuild?

If we cause a syntax error in bindings/prql-js/src/lib.rs then the playground should fail to build...

On reconsideration, I deleted my earlier comment and am replacing with this...

... changing the rust code and running task run-playground will not cause a rebuild...

I think I understand your concern, which also is part of the reason that the npm install... has the  --install-links=false option.
It sounds as if this is the desired workflow:

Edit a rust file
Run a task to compile
task run-playground to try out the result. That must pick up a changed version of the rust compiler...

If that's the case, it seems that the install-playground-npm-dependencies task could also look at the rust compiler build product to decide when to run the full npm install...

Am I on the right track?
Would you suggest a rust file to change, and the task to run to re-build it?
Any thoughts about the build product from Rust that would be a good trigger?

Thanks.

Edit a rust file

Run a task to compile

task run-playground to try out the result. That must pick up a changed version of the rust compiler...

I don't think that folks will expect two commands after editing a rust file. task run-playground should run the current code, regardless of when it was previously run. Currently this is done by the playground build calling into the prql-js build.
So either we need to either a) give that task knowledge of all the files it should check (probably crate/ and bindings/...), or b) have the JS build step know about what files it should check.
I think the latter would be great but would require redesigning the JS build with something like webpack.
The former is possible — if it provides a great speed-up then we could add it in. If it's only marginal, I would vote against, since if paths change and those become incorrect, it could be quite a confusing development experience.

I see your concern and need to understand your workflow better, but don't have time now. I'm marking this a draft and will come back next week.

I've thought about this some more. Can we go over the motivating case for the main part of this code? (TBC, some of this is just better than the old code, so keen on it overall)
Currently when developing the playground

If we change the prql compiler code, we need to re-run npm install, which then rebuilds prql-js. This took 59s on my machine (though I have prql-compiler dependencies cached), so isn't fast, but is tolerable.
If we only want to change playground code, then we get hot reloading after running npm start. This took 9s to start on my machine, and then immediately reloads on any JS change. It pick up any prql compiler code changes.

I'm very up for speeding up dev loops. But what's the thing we're trying to speed up here — the first of these? How often are we running that?

I have been thinking about workflows as well. I came to this PR to speed the second of your points: I only work on the Book or the Playground (or website, but I haven't touched that in a long time.)
This PR resolves my personal use case: run-book, run-playground, run-website now check to see if npm dependencies have changed and only run npm install when required. Changes to the pages of the Book/Playground/Web source don't require this since they rely on hot reloading, etc.
But it doesn't address your concern: making changes to the prql compiler code and then trying the changes in the Playground.

I don't think that folks will expect two commands after editing a rust file.

Never having worked on the rust code, I need to ask a naive question: What command(s) do you use with the current Taskfile.yml to edit a rust file and check it in the Playground?
I'm wondering if there's some condition that could be added to the install-playground-dependencies task (in this PR). It currently uses sources/generates to check the package.json, but perhaps it could also look at the date of the prql compiler binary. Thanks.

I have been thinking about workflows as well. I came to this PR to speed the second of your points: I only work on the Book or the Playground (or website, but I haven't touched that in a long time.)
This PR resolves my personal use case: [~run-book~], run-playground, run-website now check to see if npm dependencies have changed and only run npm install when required. Changes to the pages of the Book/Playground/Web source don't require this since they rely on hot reloading, etc.

OK, so the current commands work OK for you, it's just that the structure could use a refactor — is that right?

Never having worked on the rust code, I need to ask a naive question: What command(s) do you use with the current Taskfile.yml to edit a rust file and check it in the Playground?

task run-playground!

I'm wondering if there's some condition that could be added to the install-playground-dependencies task (in this PR). It currently uses sources/generates to check the package.json, but perhaps it could also look at the date of the prql compiler binary. Thanks.

That would be this option, quoting from above:

a) give that task knowledge of all the files it should check (probably crate/ and bindings/...),

Slowly, slowly my knowledge about the project expands... You said:

... we need to re-run npm install, which then rebuilds prql-js ...

I think you mean that you edit the rust code, then compile it, and then need to run npm install to incorporate that new prql compiler binary (prql-js?) into the Playground. True?
If so, what would the install-playground-dependencies task use for the sources and generates section? How do crate/ and bindings/ fit in? And how would I test that they were working? (You suggested creating a syntax error in a rust file, but then... what command(s) would I use to trigger the rebuild?)
Thanks for your help

I think you mean that you edit the rust code, then compile it, and then need to run npm install to incorporate that new prql compiler binary (prql-js?) into the Playground. True?

Yes, thanks for being more explicit than me!

If so, what would the install-playground-dependencies task use for the sources and generates section? How do crate/ and bindings/ fit in? And how would I test that they were working? (You suggested creating a syntax error in a rust file, but then... what command(s) would I use to trigger the rebuild?)

The crates/ & bindings/ paths (from the root of the repo) could be sources
Then make a change like adding asdf in https://github.com/PRQL/prql/blob/9c6ea2a2fa3eea2597b8596c4231f46036a30f4b/crates/prql-compiler/src/lib.rs, and run task run-playground, and it should raise an error. Currently, running npm install from web/playground will pick this up (which is the command called by task run-playground). Does that make sense?

Ahah! I understand now. (I had not realized that the wasm rebuild was part of the npm install - that output got scrolled off the screen by subsequent steps...)
And in fact asdf is a very potent poison in the lib.rs file. I see a path now. Thanks again.

@max-sixty I also see the new commit that removes --install-links from the npm install... command. Is that OK now? And should I remove it in my branch? Thanks again

@max-sixty I also see the new commit that removes --install-links from the npm install... command. Is that OK now? And should I remove it in my branch? Thanks again

Yes you can merge main in to yours

I think I have it. Any changes to package.json or to rust files in prql/crates or prql/bindings will trigger a full npm install (without the --install-links option) when you execute task run-playground.

Actually, I missed out the changes for the "check brew dependencies" - I will add those back tomorrow morning

I tested this locally and it works well!

I didn't want to delay the excellent changes, and I didn't think I was being clear on one point — so merging and will open a quick PR for your review!


Disposing an non-existing image won't fail

Currently disposing an non-existing image fails. Nevertheless the assertion that we want to be clean after disposing the image is true. Therefore we think it is nice to let a dispose of an image succeed  even if the image is not present at all.

Coverage decreased (-0.14%) to 86.94% when pulling e3943304b829e86c892fbb9edda05c8bb1c1be1a on leifhanack:failsafe-dispose-inspect into d1406732e041d9ac0bfa75ccd35a6cceb157ae7e on gesellix-docker:master.


You cant connect to most servers with this on bedrock

You cant have the same listener and connect port and the only working one is 19132 so you would need to find servers that don't use it

You cant have the same listener and connect port

That should work fine?
Also for now Bedrock servers need to have Xbox auth off and probably need to be on LAN

I only get Lan when it is on port 19132 is my problem I tried to connect to servers using it and it would just not work and it may also have to do with servers updating to 1.17

I fixed all my issues with versions and made a private test server that is pocketmine 1.16.220 it has all auth verification disabled that  I could I did the thing with bluestacks as for me at least I cannot connect to a server hosted on the same device so did that and when I join it does actually try to join how it should but the proxy does not send something right and console of pocket mine says "Failed to verify keychain link signature." And in the proxy packet log, it says "{messageSkipped:false,kickMessage:"Invalid session. Reason: Failed to verify keychain link signature."} " So I think you should add a thing where a user can override and send premade chain data with the login packet and this error should stop happening


Add BoundPredicateVisitor trait

This trait is used for BoundPredicate visitors that evaluate the predicate, and associated tests. It has been broken out of the https://github.com/apache/iceberg-rust/pull/241 PR as that was getting too large.

I've added a test for visit_op, as well as some doc comments.
Also, based on our discussion in the comments, it seemed prudent to add the non_exhaustive marker to PredicateOperator? PTAL @liurenjie1024, @marvinlanhenke, @ZENOTME as I think this is pretty much ready now

Close by #334


align with the new metadata in tensorflowin

changes in new metadata only related sparse tensors, the changes make the meaning of each sub-field consistent with dense tensors.

remove the sparseVector type
the dtype of a sparseVector will be float(the type of the element in the values field)
the shape will be the dense shape of the sparse vector
the cardinality for sparseVector will be null

Thanks Xuhong! Will help to take a look at this PR later today. :)
Why does the test fail?

@cyzhangchenya will update it, after rebasing there are some errors.


Discarded Frames

Hi,
Is there a way to detect upload speed in your library to avoid "discarded frames"?
thank you!

You have bps in this callback:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/app/src/main/java/com/pedro/rtpstreamer/customexample/RtmpActivity.java#L341
Also you can do automatic adaptative video bitrate to avoid it:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/wiki/Adaptative-video-bitrate.

Hi
public void onNewBitrateRtmp(final long bitrate) is not returning the correct bitrate. I got some value but it says that "frame discarded". Is there a way to create a listener if the frame is discarded?
btw, I'm trying to test it on poor/slow internet connection
Thank you!

"frame discarded" is not related with bitrate received in the callback.
I am counting after sending the packets to the server, therefore the discarded frames do not count.
If you want add other callback this is symple. Add a method to ConnectCheckerRtmp interface:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtmp/src/main/java/net/ossrs/rtmp/ConnectCheckerRtmp.java
Call this method after frame discarded happens:
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/blob/master/rtmp/src/main/java/net/ossrs/rtmp/SrsFlvMuxer.java#L1029
As I told you in my previous comment if you want to avoid that logcat you should use adaptative video bitrate


🐛 [firestore-bigquery-export] Partitioning not generating (No valid table reference Issue)

Extension name: firestore-bigquery-export
Extension version: 0.1.34
Configuration values (redact info where appropriate):

Cloud Functions location : us-central1
BigQuery Dataset location : us-central1
Project Id : pos-staging-0
Collection path : business/{bId}/items
Enable Wildcard Column field with Parent Firestore Document IDs (Optional) : false
Dataset ID : export_staging_latest
Table ID : items
BigQuery SQL table Time Partitioning option type (Optional) : MONTH
Remaining as default

Description:- This was working as expected and generating the timestamp on 16th August and when I started working on 17th August with new collections it stopped generating partitioned tables
Steps to reproduce: Tried all configurations, It's still generating a warning -> No Valid Table reference, Skipping Partitioning
What happened? How can we make the problem occur?

Try any configuration and try to generate a partitioned table, it is not working

Expected result
Partitioned table with _PARTITIONTIME or timestamp field and also add path_params in the schema
Actual result
Normal table with no partitioning
Attaching logs

Hi @Yashrocky96
From the configuration provided, I cannot see which field, you have chosen to partition.
Did you provide options for TIME_PARTITIONING_FIELD, TIME_PARTITIONING_FIRESTORE_FIELD and TIME_PARTITIONING_FIELD_TYPE.

In addition, partitioning cannot be added to an existing table. Could you check if this is the case for your scenario?

I have checked that the table was non-existent before I was generating. I have debugged the code to a point that when addPartitioninigToSchema is failing even when a proper class valid reference had been provided.
I have worked all configs, etc. It was working fine for 1 day and the next for no reason, It was not working.
I used the import script and updated the code to do partitioning from the code side, I have directly added options instead of calling addPartitioningToSchema function.

@dackers86, this looks similar to https://github.com/firebase/extensions/issues/1577. I have been able to recreate issue where table doesnt exist prior. In the logs I see
BigQuery dataset already exists:
Creating BigQuery table:
No valid table reference is available. Skipping partitioning
Cannot partition an existing table

which shows that its creating the table but then says cant partition an existing table

We've released a new version with a fix related to the timestamp field https://github.com/firebase/extensions/pull/1794.
Could you please try updating to the latest version 0.1.37 and let us know if the issue still persists.

Should also be resolved with #1923.


Typeahead:selected event

Hello,
How can I bind a custom function to the 'typeahead:selected' event, when a user select the "typeaheaded" suggestion with the "tab" key?
I've tried
$(addressPicker).on('typeahead:selected', function (event, result) {
    console.log(adresse + ' > typeahead:selected');
});

But no luck..

me tooo!

You can bind it on the address element itself:
$('#address').bind('typeahead:autocompleted', function (event, result) {
    console.log('address > typeahead:selected');
}

I didn't managed to get the setResult working with it though, if someone know how to do...

You can bind it on the address element itself:
$('#address').bind('typeahead:autocompleted', function (event, result) {
    console.log('address > typeahead:selected');
}

I didn't managed to get the setResult working with it though, if someone know how to do...

Maybe this helps: https://github.com/sgruhier/typeahead-addresspicker/issues/36


Drain Package Functions Should Use PollUntilContextTimeout and Return Properly Typed Errors

Drain package function waitForDelete uses wait.PollImmediate. That function is deprecated and instead PollUntilContextTimeout should be used.
In addition, we cannot check the returned error from waitForDelete function with wait.Interrupted because the function returns an error with fmt.Errorf.
What would you like to be added:
Instead of using wait.PollImmediate, we should switch to PollUntilContextTimeout . Also, the waitForDelete should return an error of type ErrorInterrupted.
Why is this needed:
wait.PollImmediate is deprecated. The current returned error is hard to check and requires string comparison.

I am happy to work on this, if this change sounds reasonable.

/triage accepted
We would accept a PR for this, thank you for offering to help with this @ugrkm

Hi, if no one is working on this, I'd like to take it up.

/assign


Goal hiding region mechanism is not flexible enough

It does not allow to make the second goal in the attached level to behave naturally. Level to reproduce: level-01.svg.txt

I need to keep everything simple. Making this mechanism more flexible introduces unnecessary complexity. I'm probably doing something wrong.


Implement plugin architecture

For #15

[x] Restruct Task/Target namespace
[x] Bundled Task/Target change to default plugin
[x] Plugin register and lookup mechanism

Coverage decreased (-0.005%) to 96.677% when pulling 34042ce50612efa212550b6163dc557bd3822513 on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-1.4%) to 95.232% when pulling 45740521bdf2dc1fa4d33a1593f1940dd7f9113d on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-1.5%) to 95.176% when pulling 1103ef894def2f339ea0b135fe7ce3f43fbe593a on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-1.2%) to 95.53% when pulling 0bea5cad81990a65ae756a0e01da11de3c6edff1 on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-0.8%) to 95.892% when pulling 1d3fbda9ab00a2a9719718fb65d871d3d14b8f6b on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-0.6%) to 96.106% when pulling b436622856592a95ca8909294330cdec1ada6789 on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.

Coverage decreased (-1.2%) to 95.48% when pulling 6c66b358f68e25b96b3fbe77a47112da366a3243 on support-plugin into 6b050e18287867fe42ea7da19f21ce20c4812480 on master.


Add view to display unfinished file stages

A view that can be used to display the processing progress of non top
level files inside of configuration editors.

Adapt CSS of FileStageItemView so that the view can be used
outside of FileIteView via a stand_alone CSS class.

Rewrite inputView visibility handling to use a CSS class to hide
the element. This allows views to set additional CSS classes that
can also hide the element without the inline style overriding
visibility.

Create subset collection of unfinished stages in stageProvider.

Add new FileProcessingStateDisplayView which displays
FileStageItemViews for all unfinished stages and hides itself if
there are non left.

Allow file type specific localized file stage description

REDMINE-15957

Coverage remained the same at 99.158% when pulling 86ca60b38da5f8905d7c0e8fb3f4cb7d1815323c on tf:file-processing-state-display-view into a40f5463d4a832a7bcb6e642c5267333e53dcc73 on codevise:master.


Enabling "Code first"

I propose this solution to make Effort usable with entity classes produced in a "Code First from database" context. Properties of such classes are adorned with ColumnAttribute, which makes Effort choke. The proposed solution overcomes this and solves issues #31, #13 and #10.

I used to insist against adding provider specific data types. However this problem seems to cause so much trouble for many developers, so I am going to let my vigorous resistance down.

I cannot get Effort to work with my Entity Framework 6.1.3 code first project. Must I create a edmx file from my existing database?

Hello @Hammer82 ,
If you need help, please create your own issue. It makes easier to follow.
You don't have to create a edmx file. You can find an example in Code First here: https://entityframework-effort.net/overview


flickr30k model download is pointing to coco model.

the link to flickr30k is pointing to coco model. Do you mind updating the link?

Thanks the link is
https://www.dropbox.com/s/cirzj1b2jul6yzx/flickr30k_nbt_1024.tar.gz?dl=0
will update now.


Distribution

Probably upload to npm, provide unpkg link that users can include.

Also consider jsDelivr, especially if they allow uploading as ES modules.

See https://esbuild.github.io/getting-started/#bundling-for-the-browser for suggested esbuild settings.


CompletionItem.textEdit deprecated in client vscode library, but not in server vscode-languageserver/node

I'm trying to setup VSCode completions to be handled by the server in specific instances, and completions are showing up and working just fine, but I want to replace the full string of text at the trigger/request location, rather than just insert the text at the location.
Basically, instead of object.existing_text becoming object.replacement_textexisting_text, I want to only see object.replacement_text.
The solution I found in the CompletionItem interface server-side was using a TextEdit instance to replace the entire string after figuring out the appropriate range.

That seemed to work just fine, but when writing some test cases I saw that in the client via the vscode library, the CompletionItem.textEdit option is deprecated, and it presents using range and insertText as an alternative.

That being said, CompletionItem.range doesn't seem to be present in the vscode-languageserver/node library version of the interface, so I'm a little confused about what is the correct approach here.
I tried the following setup for the server's returned object, but it didn't seem to work. Here, completionRange would be the range that covers the entire string of text that needs to be replaced, including the position at which the completion was triggered/requested.
return {
      label: `replacement text`,
      kind: CompletionItemKind.Value,
      preselect,
      detail: 'Item detail',
      documentation,
      range: {
        replacing: completionRange,
        inserting: Range.create(completionRange.start, completionRange.start)
      },
      insertText: `replacement text`
    };

The shape of the Completion item is not the same for LSP and VS Code API. The reason is some sort of backwards compatibility. The conversion of one into the other is handled here: https://github.com/microsoft/vscode-languageserver-node/blob/f02ec883a68c6ee0e07a512c2114847f5f8c9d29/client/src/common/protocolConverter.ts#L547
In our case you should stick to the LSP shape and let the LSP client do the conversion.

So does that mean it's still safe to use CompletionItem.textEdit via the LSP server as the client is converting that information?


[SPARK-38028][SQL] Expose Arrow Vector from ArrowColumnVector

What changes were proposed in this pull request?

This change exposes Arrow Vector from ArrowColumnVector.
Why are the changes needed?

In some cases we need to work with Arrow Vectors behind ColumnVector using Arrow APIs. For example, some Spark extension libraries need to consume Arrow Vectors. For now, it is impossible as the Arrow Vector is private member in ArrowColumnVector. We need to expose the Arrow Vector from ArrowColumnVector.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Existing tests.

cc @sunchao @dongjoon-hyun

I guess it's fine to expose .. @BryanCutler FYI.

Merged to master for Apache Spark 3.3.
All compilation (SBT/Maven/Scala-2.13/Java 11/Java 17) passed and this is irrelevant to the exiting UTs.

Thank you @dongjoon-hyun @sunchao @HyukjinKwon

A little late, but SGTM

Thank you, @BryanCutler .

Thank you @BryanCutler


Variable declaration list: only 1st variable colored

Problem found using vscode v0.10.2
Variables after the first in a list of declarations don't get the special color applied.

Note: Not all themes support this extra color.

VS Code 0.10.10. It works if the declaration is single-line but breaks if it's not:


upload quarkus from 1.x to 2.x

Description

Expected behavior
update quarkus from 1.x to 2.x

Hey there. Can you please be a bit more specific? What issue do you have? The current integration compiles against Quarkus 1.x, but is compatible also with 2.x, so there should not be an issue.

Closing since I received no response for 2 weeks


Using Azure App Configuration via Sprint Boot Application

I am trying to follow this documentation:
https://docs.microsoft.com/en-us/azure/azure-app-configuration/quickstart-feature-flag-spring-boot
I have set the environment variable with the required connection string to connect to Azure App Configuration. However, I am getting the following exception while running it:
(I have added spring.cloud.azure.appconfiguration.stores[0].name to bootstrap.properties file.)

APPLICATION FAILED TO START

Description:
Binding to target [Bindable@2cf92cc7 type = java.util.List<com.microsoft.azure.spring.cloud.config.stores.ConfigStore>, value = 'provided', annotations = array[[empty]]] failed:
Property: spring.cloud.azure.appconfiguration.stores[0].name
Value: ${APP_CONFIGURATION_CONNECTION_STRING}
Origin: class path resource [bootstrap.properties]:1:53
Reason: The elements [spring.cloud.azure.appconfiguration.stores[0].name] were left unbound.

Action:
Update your application's configuration
2020-04-08 01:52:29.941 ERROR 28788 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'initConnectionString' defined in class path resource [com/microsoft/azure/spring/cloud/config/AppConfigurationBootstrapConfiguration.class]: Unsatisfied dependency expressed through method 'initConnectionString' parameter 0; nested exception is org.springframework.boot.context.properties.ConfigurationPropertiesBindException: Error creating bean with name 'spring.cloud.azure.appconfiguration-com.microsoft.azure.spring.cloud.config.AppConfigurationProperties': Could not bind properties to 'AppConfigurationProperties' : prefix=spring.cloud.azure.appconfiguration, ignoreInvalidFields=false, ignoreUnknownFields=true; nested exception is org.springframework.boot.context.properties.bind.BindException: Failed to bind properties under 'spring.cloud.azure.appconfiguration.stores' to java.util.List<com.microsoft.azure.spring.cloud.config.stores.ConfigStore>
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:539) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6181beec-df25-e22e-fd74-e771c52ac207
Version Independent ID: e25f278a-8d64-430d-72eb-8520a5ec4d90
Content: Quickstart for adding feature flags to Spring Boot with Azure App Configuration
Content Source: articles/azure-app-configuration/quickstart-feature-flag-spring-boot.md
Service: azure-app-configuration
GitHub Login: @lisaguthrie
Microsoft Alias: lcozzens

@gkgaurav31 Thanks for the question! We're investigating this and will get back to you shortly.

@gkgaurav31 Please make a correction in your bootstrap.properties file as follows:
spring.cloud.azure.appconfiguration.stores[0].connection-string= ${APP_CONFIGURATION_CONNECTION_STRING}
and then build and run your application. Navigating to http://localhost:8080/welcome should pull up the output as described in the doc.
I'll correct the documentation to reflect this change.
Thank you very much for bringing this to our attention!

@gkgaurav31 I have submitted a PR to fix the issue that's awaiting review.

@gkgaurav31 The change is merged internally and will go-live in the public docs shortly. Please let me know if you see issues.

i have the same issue but in my bootstrap.properties my connection endPoint is = spring.cloud.azure.appconfiguration.stores[0].connection-string=Endpoint=https:------------'
and my pom is this:

org.springframework.boot
spring-boot-starter-parent
2.2.4.RELEASE

4.0.0
<artifactId>azure-appconfiguration-sample</artifactId>
<name>Spring Cloud Azure App Configuration Sample</name>
<dependencies>
    <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>spring-cloud-azure-appconfiguration-config</artifactId>
        <version>1.2.2</version>
    </dependency>
</dependencies>


Suite name

Currently, we refer to this suite as effect-handlers-bench. For marketing purposes it would be good to come up with a more felicitous name. Let's collectively think about good names; here's some examples to bootstrap the process:

MeasureFX
BenchFX (I am using this name now for a different project: https://github.com/wasmfx/benchfx)
HaRP (Handler Runtime Performance)
EHOPE (Effect Handler Oriented Performance Evaluation)
FXmark

... we should have a logo too!

@jasigal lets call a vote on the name.

PerFX (or something related, think "perfection" 😆)
HaRPe (same but with the "e", pronounced harpy like the mythical creature)

from daniel: HaRPE, E =evaluation


Resolved the issue 2244 now links are opening in a new tab

PR Description 📜
Please include summary related to the issue you have fixed and describe your PR in brief over here by specifying the issue number on which you were working below
Fixes # 2244
Now the links are opening in a separate tab

Mark the task you have completed ✅

[ ] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[ ] I have performed a self-review of my own code or work.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] My changes generates no new warnings.
[ ] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[ ] I have added screenshot for website preview in assets/images
[ ] I have added entries for my game in main README.md
[ ] I have added README.md in my folder
[ ] I have added working video of the game in README.md (optional)
[ ] I have specified the respective issue number for which I have requested the new game.

Add your screenshots(Optional) 📸

Thank you soo much for contributing to our repository 💗

Thank you @abansal15 ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

Hey @abansal15,
Please make sure to link the relevant issue using the appropriate syntax, such as "#issueNumber" 👀.
Follow the proper guideline and make a new PR again 😀.
Happy Hacking 💗

Thank you  @abansal15 , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀

hello how can i now is my pr is merged or not as this issue is closed ?


[docs] Fix syntax error in migration-v3-ru.md

[ ] I have followed (at least) the PR section of the contributing guide.

Translated docs are handled in crowdin.