Prototype for a defragmentation interface that supports tiling optimal images

Hey Adam,
First of all, let me preface this by saying that this is a incomplete pull request for now! But I'm very interested in your feedback on this and to see if you are interested in having this upstream.
Essentially this is the foundational work for defragmentation that supports tiling optimal images. The basic idea is that there is a new defragmentation info VmaDefragmentationInfo3, which has support for callbacks that the host can fill in. These callbacks are then used to figure out what kind of allocation something is and other information, such as the create info and barrier information. The only other visible difference is that there now is an vmaDefragmentationBegin2() to go along with it.
My goal was to make this as little intrusive as possible, all of the old schemes still work and they work the exact same way as before. Almost none of the public interfaces have been touched, and I tried to also minimize the code changes internally.
When defragmenting, VMA still does all the same work to figure out where to move new allocations. The big difference comes at execution time (see ApplyDefragmentationMovesGpu). Here, VMA will interrogate the host in order to figure out where the resource is used and what image layout it is in. Then VMA will create a new image, bind it to the new memory location, and do the copying & pipeline barriers necessary to synchronize everything. Because of the way images are handled, creating and binding a new image is also no longer in the responsibility of the host application, but they are provided with another callback to let them know that a resource has changed.
For an example of this, take a look at the modified Tests.cpp which has an TestDefragmentationImagesGpu() function that tests the new interface.
There are a bunch of open issues that I'm left with:

[x] No support for buffers. My thought is to provide the exact same interface as for images, and most of it is already there, except for the execution part
[ ] There is a callback to interrogate the host about the type of resource, but VMA already knows this internally from the VmaSuballocationType. This mainly was my idea to make this an opt-in interface, but I'm not sure if this makes for a better or worse API.
[ ] Because of the way it works, it requires a command buffer. Which is at odds with how defragmentation works on the CPU. Then again, tiling optimal images might not really be something you ever have on the CPU, so it might be okay to have this be GPU only?
[ ] Documentation! VMA is great because it has excellent documentation, and I documented basically nothing yet.
[ ] Bit of clean up and whitespace issues

Anyways, my question is essentially: What are your thoughts on this? Would you be interested in this upstream? And if so, what are your thoughts on the API proposal?

Also I guess another point that is missing is not having the CI fail :P

I actually have some more thoughts on this... And I'd love to hear your thoughts on this as well.
I've been reviewing our requirements, and the stop the world approach to defragmenting imposes a huge burden on us. The whole goal with Vulkan is that we never have to stop, we have pre-compiled all state and know when our memory is fully specified. However, with the current way defragmenting works, it requires us to stop using all our assets, make sure the GPU is done using them, then queue the defragmentation, wait for that to complete on the GPU and retrieve our new objects. The latter can be slightly improved with the new callbacks in this pull request, where objects can become known to the host earlier and all that's needed is a pipeline barrier to sync it all up. But the other big showstopper is that while the defragmentation is running, no new objects can be made unless you put them into a new pool.
 Essentially a multi step defragmentation process. First it'd need to generate a defragmentation plan with moves and blocks that will become empty and need deletion in the future. Those blocks would need locking so that no further allocations will be made from them, but leaving the other blocks open for new allocations (including making new blocks). This would happen in a vmaDefragmentationBegin() equivalent with the plan saved in the defragmentation context.
Next up would be the incremental part, something like vmaDefragmentationContinue(). The library will create new allocations for as many moves as it possibly can before running out of space, then move them all over. It also tells the host about the new allocations so it can update whatever it needs to update (invalidate descriptor sets, create new image views, what have you). The application will then have to continue running using the new allocations, while the GPU might still be working with the old data. Once the application knows the GPU is done with all of the swapped out allocations, it will call vmaDefragmentationContinue() again, which will delete the old allocations that have been replaced and continue moving things around. Here it can also already delete empty memory blocks. This process would repeat until the plan finished execution. Of course, since the pools aren't locked, the execution plan needs periodic updating because of other allocations that changed.
vmaDefragmentationContinue() would return VK_NOT_READY when it needs to be invoked again, and VK_SUCCESS when it's done. Of course this requires multiple command buffer submissions and possible running across many frames. But the advantage is that there is no requirement to stop the world and defragment the memory. All that is required is to have a pipeline barrier after every delete/move pair, which could in theory be fine grained or just transfer stage to top of pipe.

Hello,
First of all, thank you for opening this pull request. That's the biggest contribution to the library I got so far! I'm willing to upstream your code to the library and also work on it (subject to my time constraints) as decent defragmentation is a very needed feature. Right now I plan to release VMA version 2.3.0 as-is in December 2019, but this defragmentation can become a big thing for the next release.
I'm curious what's your use case. Are you developing a game or some other real-time, interactive graphics application? For which platforms? It will be great if your defragmentation implementation is field-tested in a real large scale app before it gets to the library. Designing API for it is also better approached from user's perspective, thinking about what API would be convenient to use in the app.
I analyzed your code changes and following comments come to my mind:
Image/buffer creation is something I would rather leave to the user. You designed a callback that queries for VkImageCreateInfo, calls vkCreateImage and vkBindImageMemory inside of the library. I think that's not a robust solution. Think what happens if the user wants to attach custom structures to VkImageCreateInfo::pNext or use vkBindImageMemory2 and provide custom pNext while binding, plus lifetime management of those additional structures.
Same with barriers. It's hard to make them inside of VMA in a correct and robust way. Looking at current implementation, consider following issues:

beginImageBarrier for newly created image has dstAccessMask = 0. This doesn't look correct.
What if a resource has VK_SHARING_MODE_EXCLUSIVE not CONCURRENT, was used on graphics queue, we want to defragment on transfer queue, and so it needs queue family ownership transfer barrier? Passing VK_QUEUE_FAMILY_IGNORED is not correct is this case.
Vma*Introspection::accessMask, stageMask - what do they represent? The last access before defragmentation? Planned next access after defragmentation? I can see it's both. What if I want the image to be accessed differently before and after defragmentation?

Callback functions you designed would do the job, but I would prefer to avoid such "push approach" and design something more like a "pull approach" where the user needs to call some VMA functions repeatedly, interleaved with doing his required job. That's why I also agree with you that "stop the world" approach is bad and it would be better to provide some way to defragment incrementally, using some vmaDefragmentationContinue function you proposed. I like that a lot.
Even better if the defragmentation process doesn't block other allocation/free operations on the same memory pool performed in the meantime. But this is something very difficult to implement, so I tired to avoid it as long as I could. It will require keeping some additional information in memory blocks and/or allocations to be used also during normal allocation/free. I think the approach you proposed - to designate some memory blocks to become empty after defragmentation - is the easiest and the best one. Nonetheless, it will probably require some significant refactor or rewrite of the whole defragmentation code. I'm ready to get rid of the current solution entirely in favor of this new one.
My rough idea for the API is that a function to be called repeatedly to perform a single defragmentation step would:

take "defragmentation context" object, maxBytesToMove, maxAllocationsToMove
calculate what to move and where
reserve memory for the new copies
fill in an user-provided array with details of allocations to be moved in that step, along with VmaAllocation objects.

It would be user's responsibility to:

fetch VmaAllocationInfo::pUserData if needed to deduce what resource is represented by that allocation
create new buffer/image in a specific place, bind it to the specific memory
make sure current image is in TRANFER_SRC layout and the new one in TRANSFER_DST (or GENERAL), issue appropriate barrier if necessary

Then another VMA function would need to be called with given VkCommandBuffer. There, VMA would:

would record vkCmdCopy... commands to it
switch VmaAllocation objects to point to new places in memory

Then again the user would need to issue barriers that transition destination images to their desired layouts, submit this command buffer, vkDestroyImage old images (VmaAllocation objects stay the same), and make some final call to VMA when finished so that it can free their old memory.
This way, storage of VkImageCreateInfo etc., as well as barriers stay outside of VMA as before. What do you think about this?
Some minor comments about the current code:
VmaAllocationType is probably not needed as we have this information already.
I don't like ImageAspectMaskForFormat. How do we make sure we support all the possible formats? New ones are added with Vulkan updates. Besides, COLOR is not the only aspect possible when not working with depth/stencil. How about planar textures? I would try to avoid requiring this function entirely.
Returning large structures like Vma*Introspection from functions by value is considered a bad practice by many developers. Such data records are usually returned by output (pointer) parameter. That's how Vulkan and VMA do it as well.

Thank you so much for your feedback!
As far as our use case goes, I work at Laminar Research on the Vulkan port of https://x-plane.com, a professional flight simulator. We have a planet sized streaming world, that users can put additional things in such as ortho scenery to further increase memory pressure. Our two big main concerns in terms of fragmentation is the streaming of new bits of the planets (which come in as 1x1 degree chunks) and general overcommitment of memory where X-Plane has to decide at runtime to ditch higher mip levels of some textures. We are quite good at keeping within the budget right up until the moment the budget itself shrinks. In which case we have to start dumping memory, and our best bet for that is to let VMA defragment our pools. In my stop the world tests with the above code, I can easily free 500mb of memory blocks and get the driver a good chunk of memory back. However, on the other hand, our big use case for Vulkan is that we want a stutter free world. All pipelines are pre-built and all assets are accounted for before we render a frame with them, to make sure that we never have to wake up mid-frame and block anything. Right now the engine uses OpenGL, so that's been a constant problem for us. As far as platforms are concerned, we are running Vulkan and Windows and Linux (macOS uses Metal directly, not MoltenVK) and in the future maybe Android, although we have much better control over allocations there.
Re Buffer/Image creation: The point about VkImageCreateInfo::pNext is excellent (you can tell that our needs for fancy image creation is minimal). But it would indeed be much better to punt that to the user instead.
dstAccessMask = 0 is I believe okay because the image is in undefined layout. No need to invalidate a cache when there's no data to invalidate?
With respect to queue ownership, my thought was that the resource you want to defragment needs to be owned by the queue that owns the command buffer. So no ownership transfers necessary as far as the library is concerned. Otherwise you'd have to pass in both queues and then it becomes even more of a headache because what if the queue doesn't even support the layout transitions necessary? So ideally my thought was that the queue ownership doesn't change, and you do this on a queue that supports both transfer + whatever your image currently needs. Same of course goes for buffers, but without the headache of layout transitions.
Vma*Introspection::accessMask, stageMask represents the queue to either wait on, so assuming a non stop-the-world defragmentation and a read only resource, this would realistically be top of pipe. Or whatever pipeline last wrote to it, so that the write doesn't run out of order with the defragment. However, it servers as a double (and maybe stupid) role, by also defining the dst stage in the reverse case. So after it's moved, this will be the pipeline stage that is unblocked from the transfer stage. So this might become an OR of two stages. Although thinking about this some more, breaking them out into src and dst might be a much better thing to avoid unnecassry GPU pipeline draining.
Re Callback functions: You should've seen my first idea with lambdas :P
All kidding aside, I see where you are coming from. My idea was to make it a little bit simpler to swap out allocations, instead of having to retrieve them a second time later on. But the runtime guarantees are kinda iffy and it might make more sense to have the user bulk poll them. However, I think they are necessary for VMA to pull data, since it'd be non-ideal to have the user create a bunch of new resources that VMA might or might not use. Right now the advantage of the callbacks here is that the user only needs to provide data when VMA decided it actually wants to move the allocation. This would be even more of an advantage if resource creation is punted onto the user.
I agree with you about the lock-free-ish part of the defragmentation, I've spent the weekend trying to think of a good solution to this. I think a simple per block locking might be a somewhat good solution? When a resource allocation is requested, VMA would try to take the lock for a block and otherwise move on to a different block. Of course, I can also see the big downside here where multiple cores are trying to allocate things at the same time. Maybe some kind of spinning on the lock might be an appropriate middle ground here? What I'm thinking is that if the locking is on a per block level, then during defragmentation, VMA could just lock off specific blocks that it wants to free later on.  Threads could always be additionally funnelled to avoid extra block creation when multiple threads start fighting.
With your proposed API, I do like the general idea of it, but I have one thought about the barriers. My hope with moving the barriers into VMA was that they can be minimized. VMA knows what to move, so it can issue just barriers for allocations that need it and everything in place can stay there (my use case is to throw everything and the kitchen sink at VMA in terms of defragmentation and then let it figure out what to do).
If the user has to do the barriers, then the API would need to be something like this (I believe. Terribly named API ahoy):
VmaDefragmentationSomeResultStructure result;
vmaDefragmentationFigureOutWhatToMove(&result);

vector<MyAllocationThing> = figureOutWhatMoved(result); // List of allocations that are going to move

// Pipeline barriers here

vmaDefragmentationActuallyMove(&result);

// Other pipeline barriers here

So it'd be a two step process. Although the advantage would be that the user has much greater visibility into what state their resources are in at very specific times. So this might be an advantage after all? Rethinking this, I don't feel as strongly as I thought I would about where the barriers happen. Also, it'd get rid of the other callbacks necessary for resource creation, because where the first barrier happens, resource creation could also happen. And where the second barrier happens, the user can swap out their old resources for the new ones.
Lastly:
VmaAllocationType I agree. It's my poor mans opt in mechanism, but realistically the opt in would be actually using the new API.
ImageAspectMaskForFormat Yyyyyyep... That one is a bit of a weird one.
Vma*Introspection I tend to trust the compiler with regards to copy elision. I know it's not considered the nicest style for most. But I think that problem might solve itself if the user has to actually create all the resources and the callbacks go away?

I think they are necessary for VMA to pull data, since it'd be non-ideal to have the user create a bunch of new resources that VMA might or might not use. Right now the advantage of the callbacks here is that the user only needs to provide data when VMA decided it actually wants to move the allocation. This would be even more of an advantage if resource creation is punted onto the user.

I can't see why a two-step process would be needed. With the "pull" approach the library could also return for each step a request to recreate only those resources that are really going to be moved in that step. Sames goes with barriers. My idea is that this VMA call would return ("fill in an user-provided array", as I said) only the list of resources to be moved. So the variable VmaDefragmentationSomeResultStructure result you mentioned would already contain the list of moves to be peformed at that step, not all the resources available for defragmentation, as passed to the defragmentation begin function. Does that make sense? I'm not sure if I explained it clearly... Your code is:
VmaDefragmentationSomeResultStructure result;
vmaDefragmentationFigureOutWhatToMove(&result);
vector<MyAllocationThing> = figureOutWhatMoved(result);

For me it's more like:
const uint32_t maxAllocationsToMove = 128;
vector<VmaDefragmentationCopy> defragCopies(maxAllocationsToMove);
uint32_t defragCopyCount;

// This function:
// - calculates list of moves to be performed in this step
// - reserves memory for destinations
// - fills defragCopies array
// - may already switch VmaAllocation objects to new location
vmaDefragmentationStepBegin(defragCtx, maxAllocationsToMove, &defragCopyCount, defragCopies.data());

for(uint32_t i = 0; i < defragCopyCount; ++i)
{
    const VmaDefragmentatonCopy& copy = defragCopies[i];
    EngineTexture* myTexture = FindEngineTexture(copy.allocation);
    vkCreateImage(vkDevice, myTexture->imageCreateInfo, &myTexture->newImage);
    vkBindImageMemory(vkDevice, myTextyre->newImage, copy.dstDeviceMemory, copy.dstOffset);
    vkCmdPipelineBarrier(cmdBuf, myTexture->newImage, UNDEFINED -> TRANSFER_DST);
    vkCmdPipelineBarrier(cmdBuf, myTextyre->currImage, SAMPLED_IMAGE -> TRANSFER_SRC);
    vkCmdCopyImage(myTexture->currImage -> myTexture->newImage);
    vkCmdPipelineBarrier(cmdBuf, myTexture->newImage, TRANSFER_DST -> SAMPLED_IMAGE);
}

vkCmdSubmit(cmdBuf)
// Wait for cmdBuf to finish.

// This function frees memory of source images.
// (Maybe it could make sense to require user to pass returned data again to avoid keeping internal copy of it.)
vmaDefragmentationStepEnd(defragCtx, defragCopyCount, defragCopies.data());

for(uint32_t i = 0; i < defragCopyCount; ++i)
{
    const VmaDefragmentatonCopy& copy = defragCopies[i];
    EngineTexture* myTexture = FindEngineTexture(copy.allocation);
    vkDestroyImage(vkDevice, myTexture->currImage);
    myTexture->currImage = myTexture->newImage;
    myTexture->newImage = VK_NULL_HANDLE;
    // Recreate views and switch descriptors to point to myTexture->newTexture.
}

I included vkCmdCopyImage as an operation performed by the user as well beacuse after some thinking I concluded this is a better way to go, as this call requires information that VMA doesn't have, like texture extents.
dstAccessMask = 0 I'm not 100% sure about that either (Vulkan barriers are hard...), but I think going from UNDEFINED doesn't justify specifying 0. This flag must represent the intended usage after of the resource after the barrier and must also match pipeline stage mask. Using UNDEFINED as source layout doesn't necesarily mean no-operation in the driver. It may initialize some metadata of an internal texture compression format that would otherwise cause GPU to crash if left garbage or not synchronized properly.

With respect to queue ownership, my thought was that the resource you want to defragment needs to be owned by the queue that owns the command buffer. So no ownership transfers necessary as far as the library is concerned. Otherwise you'd have to pass in both queues and then it becomes even more of a headache because what if the queue doesn't even support the layout transitions necessary?

But if the resource is used on a different queue than where we want to defragment, it would be best to do the queue family ownership trasfer barrier at the same time as we transition the image to the required TRANSFER_SRC layout. That's why I would prefer the barriers to be done by the user.
About locking of memory blocks: Yeah, that sounds fine. If the 1st of the calls to be made in each step reserves memory for move destinations, and memory for move sources stays reserved as before, then affected memory blocks don't even need to be excluded from parallel allocation operations for the time the defragmentation step is performed. The only thing we need is to mark source block with a special flag telling that we want to make it empty so no new allocations are made out of it. The only difficulty is the question of what additional data need to be kept in Metadata_Generic and/or VmaAllocation_T during the defragmentations step.

I've prototyped some more on this, the old callback based system is gone. The new incremental model makes use of the defragmentation flags to signal that it's being used by the user. The basic usage would be:
VmaDefragmentationInfo2 info = {};
info.flags = VMA_DEFRAGMENTATION_FLAG_INCREMENTAL;
// Fill the rest in here
// However, no command buffer is required here!

VkResult result = vmaDefragmentationBegin(m_allocator, &info, &stats, &context);
if(result == VK_SUCCESS)
	return; // No work to be done

std::vector<VmaDefragmentationStepMoveInfo> all_moves;
all_moves.resize(...);

result = VK_NOT_READY;

while(result == VK_NOT_READY)
{
	VmaDefragmentationStepInfo step_info = {};
	step_info.moveCount = all_moves.size();
	step_info.pMoves = all_moves.data();

	vmaDefragmentationStepBegin(m_allocator, &step_info, context);

	VkCommandBuffer command_buffer = ...;

	// Iterator over the moves indicated by step_info and execute them

	vkQueueSubmit(command_buffer, ...);

	// Start using the newly created resources in future command buffers!

	// Wait until the command buffer finishes executing and all
	// old resources are being stopped in use.
	// Realistically this doesn't really happen like this but is invoked
	// as result of a callback or something

	result = vmaDefragmentationStepEnd(m_allocator, context);

	// Now delete all of the previous resources.
	// VkImage, VkBuffer, VkImageView etc
}

vmaDefragmentationEnd(m_allocator, context);

There currently is no locking of the memory blocks, so concurrent allocations could squeeze in and prevent the blocks from being freed. Although in my tests with heavy memory pressure it worked out to be eventually consistent with what I expected to happen. In general, because vmaDefragmentationStepBegin() can't free the old resources yet, for a brief moment until vmaDefragmentationStepEnd() is called, there is heavier memory usage within the app. Although this one can't really be fixed.
I do plan on revisiting the memory block locking though, and maybe trying to see if the whole thing can be a bit cleverer. Incremental defragmentation also requires the generic defragmentation algorithm, since fast moves within a block aren't readily supported. Given the existing other constraints, that might not be an issue either though, since I don't think the fast algorithm would be used a lot.
And lastly, visualizations! I've excluded dedicated allocations, but just to show that it works in a real application. The usage before defragmenting was roughly 6.61Gb as reported by the memory budget. After it's down to 5.81Gb due to memory compacting. This was with heavy background allocations, as well as running completely on a background thread without interfering with the main thread rendering:
Usage before defrag:

Usage after defrag:

Thanks for this contribution. It looks great! I will look more deeply into the code and hopefully merge it when I find more time (I have little time to dedicate to this project recently), so for now just minor comments:
"DefragmentationStep" - we discussed before that maybe it would be better to call it "DefragmentationPass", for consistency with Vulkan.
vmaDefragmentationStepBegin - I would rather specify context as second parameter and step_info after that.

Glad you are liking it! No worries with regards to time, I'm in the same boat right now with Christmas and deadlines looming.
I noticed you committed the changes in a separate commit, I was going to make this pull request mergeable once I've actually written up the documentation and fixed the locking issue and the naming from step to pass. If you want, I can do these in new pull requests (since I'm not sure this one will still merge properly?), or if you'd rather do that, you are also very welcome to do so :)

I've merged your change and then made additional commit that: 1. renames "step" to "pass", 2. fixes tests, as they were not correctly deleting old VkBuffer/VkImage and switching pointer to new one in AllocInfo.
Again thank you very much for this significant contribution! Please feel free to propose any further changes you like. Things that come to my mind at the moment are:

It would be good to enable call to ValidateGpuData in the tests for new defragmentation, to validate the contents of textures.
I can see you've added methods TryLock, TryLockWrite, TryLockRead, and used one of them once. Is it really needed?
I would like to add some parameters maxBytesToCopy, maxAllocationsToCopy to defragmentation step.
Finally, maybe the whole API of the defragmentation feature is worth revisiting to completely remove/deprecate the old way of defragmenting using memmove. That would probably simplify the internal implementation.


Setting content type on file isn't working

I've modified this sample to include the content type so the file(s) are understandable by a browser, but this doesn't set the content type:
   var options = { contentSettings: { contentType: 'application/pdf'}};     fileService.createFileFromLocalFile(shareName, directoryName, fileName, imageToUpload, options, function (error) {     	if (error) {     	  callback(error);     	} else {
This doesn't work with a PDF and it doesn't work with an image (image/png).  How can I set the content type on an uploaded file so the file is accessible from my website?

@seguler Can you comment?


ODI DFP-34X-2C2 ONT and possible issue with VLAN operation instructions

I'm using ODI DFP-34X-2C2(firmware version V1.0-220923) ONT as a replacement for the ISP-provided Huawei HG8010H ONT. The ODI DFP-34X-2C2 ONT is authenticated and in the operational state:
# diag gpon get onu-state
gpon get onu-state
ONU state: Operation State(O5)
RTK.0> command:#
#

Same can be seen from the web interface:

According to omcicli mib get 84, the OLT has assigned the VLAN tagging filter data:
# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x01
FilterTbl[0]: PRI 0,CFI 0, VID 1
FilterTbl[2]: PRI 0,CFI 0, VID 4
FwdOp:  0x04
NumOfEntries: 2
=================================
=================================
EntityID: 0x02
FilterTbl[0]: PRI 0,CFI 0, VID 4
FwdOp:  0x04
NumOfEntries: 1
=================================
#

Once I disable the VLAN tag investigation for the tagged frames with omcicli mib set 84 0x01 FwdOp 0x02(Open OMCI Specification pages 69 - 72) and create the VLAN interfaces in the router, then both the Internet connectivity(VLAN 1) and IPTV service(VLAN 4) seem to be working perfectly fine. Example from a Linux-based router:
root@r1:~# # <IP_ADDRESS>/24 is for the DFP-34X-2C2 management
root@r1:~# ip -4 a sh dev wan1
6: wan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global wan1
       valid_lft forever preferred_lft forever
root@r1:~#
root@r1:~# ip -4 a sh dev wan1.1
10: wan1.1@wan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet <IP_ADDRESS>/28 brd <IP_ADDRESS> scope global dynamic wan1.1
       valid_lft 1610sec preferred_lft 1610sec
root@r1:~#
root@r1:~# ip -4 a sh dev wan1.4
11: wan1.4@wan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet <IP_ADDRESS>/18 brd <IP_ADDRESS> scope global dynamic wan1.4
       valid_lft 14104sec preferred_lft 14104sec
root@r1:~#

Publicly routable IPv4 addresses in the output above are obfuscated and IPv6 addresses are not shown.
However, both in case of Huawei HG8010H ONT and Huawei MA5671A SFP ONT, there was no need for the VLAN interface for VLAN 1. In other words, the Internet traffic was untagged and IPTV traffic was in the VLAN 4(details are described here).
What might cause the behavior described above? Could it be that the OLT does not send the VLAN operation instructions to ONT? For example, in case of ODI DFP-34X-2C2 there seems to be no instructions:
# omcicli mib get 171
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ExtVlanTagOperCfgData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#

.. while in case of Huawei HG8010H there seems to be something:
WAP>omcicmd mib show meid 171 instid 65535

  -------------------------------------------------------
  Me(171)(Extend Vlan Tag) has 1 instances, instance size is 40
  -------------------------------------------------------
  Instance ID = 0x0001, Instance Content:
  02830000 81008100 6843d400 01010000 00000000 00000000 00000000 00000000
  -------------------------------------------------------

success!
WAP>

Try firmware M110_sfp_ODI_240202.tar

Thanks, I upgraded the firmware to V1.1.6-240202:

Unfortunately, the VLAN tagging filter data and Extended VLAN tagging operation configuration data are identical to old firmware version V1.0-220923:
# omcicli mib get 84
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VlanTagFilterData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=================================
EntityID: 0x01
FilterTbl[0]: PRI 0,CFI 0, VID 1
FilterTbl[2]: PRI 0,CFI 0, VID 4
FwdOp:  0x04
NumOfEntries: 2
=================================
=================================
EntityID: 0x02
FilterTbl[0]: PRI 0,CFI 0, VID 4
FwdOp:  0x04
NumOfEntries: 1
=================================
#
# omcicli mib get 171
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ExtVlanTagOperCfgData
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#

After upgrading the firmware to V1.1.6-240202, I enabled the OMCI_LOGFILE, imported the hex dump to Wireshark with OMCI plugin installed and finally saved the messages to pcap-ng file.
In total, there were 976 OMCI messages logged, but only 26 of those have a unique managed entity class name(OMCI specification pages 507 - 516):
martin@lab-svr:~$ tshark -r ODI_DFP-34X-2C2_fw_240202_omcilog.pcapng -T fields -e omciproto.me_class_str | sort -u
Ethernet frame extended PM (014e)
Ethernet PM History Data (0018)
Ethernet PM History Data 2 (0059)
Ethernet PM History Data 3 (0128)
FEC PM History Data (0138)
GAL Ethernet profile (0110)
GEM interworking Termination Point (010a)
GEM Port Network CTP (010c)
GEM Port PM History Data (010b)
MAC bridge port configuration data (002f)
MAC bridge port filter preassign table (004f)
MAC Bridge Port PM History Data (0034)
MAC Bridge Service Profile (002d)
Multicast GEM interworking termination point (0119)
OLT-G (0083)
ONT Data (0002)
ONT-G (0100)
Priority queue-G (0115)
Reserved for future standardization (351) (015f)
***TBD*** (78) (004e)
T-CONT (0106)
Threshold Data 1 (0111)
Threshold Data 2 (0112)
Traffic Scheduler-G (0116)
Virtual Ethernet interface point (0149)
VLAN tagging filter data (0054)
martin@lab-svr:~$

As seen above, looks like the OLT did not send the Extended VLAN tagging operation configuration data. In addition, I looked for the non-zero result code(OMCI specification pages 520 - 521) in messages sent by ONU to OLT:
martin@lab-svr:~$ tshark -r ODI_DFP-34X-2C2_fw_240202_omcilog.pcapng -Y "omciproto.msg_type_ar == 0 and frame[22:1] != 0" -O omci
Frame 58: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: Send_00 (20:53:45:4e:44:00), Dst: Receive_00 (20:52:45:43:56:00)
OMCI Protocol, ONU< Create               - Reserved for future standardization (351)
    Transaction Correlation ID: 38560
    Message Type = Create
        0... .... = Destination Bit: 0x0
        .0.. .... = Acknowledge Request: 0x0
        ..1. .... = Acknowledgement: 0x1
        ...0 0100 = Message Type: Create (4)
    Device Identifier: 0x0a
    Message Identifier, ME Class = Reserved for future standardization (351), Instance = 257
        Managed Entity Class: Reserved for future standardization (351) (015f)
        Managed Entity Instance: 0x0101
    Result: Unknown managed entity (04)
    Trailer
        CPCS-UU and CPI: 0x0000
        CPCS-SDU Length: 0x0028
        CRC32: 0x89e1a54b

Frame 302: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: Send_00 (20:53:45:4e:44:00), Dst: Receive_00 (20:52:45:43:56:00)
OMCI Protocol, ONU< Create               - Reserved for future standardization (351)
    Transaction Correlation ID: 39362
    Message Type = Create
        0... .... = Destination Bit: 0x0
        .0.. .... = Acknowledge Request: 0x0
        ..1. .... = Acknowledgement: 0x1
        ...0 0100 = Message Type: Create (4)
    Device Identifier: 0x0a
    Message Identifier, ME Class = Reserved for future standardization (351), Instance = 257
        Managed Entity Class: Reserved for future standardization (351) (015f)
        Managed Entity Instance: 0x0101
    Result: Unknown managed entity (04)
    Trailer
        CPCS-UU and CPI: 0x0000
        CPCS-SDU Length: 0x0028
        CRC32: 0x6781d909

Frame 546: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: Send_00 (20:53:45:4e:44:00), Dst: Receive_00 (20:52:45:43:56:00)
OMCI Protocol, ONU< Create               - Reserved for future standardization (351)
    Transaction Correlation ID: 40187
    Message Type = Create
        0... .... = Destination Bit: 0x0
        .0.. .... = Acknowledge Request: 0x0
        ..1. .... = Acknowledgement: 0x1
        ...0 0100 = Message Type: Create (4)
    Device Identifier: 0x0a
    Message Identifier, ME Class = Reserved for future standardization (351), Instance = 257
        Managed Entity Class: Reserved for future standardization (351) (015f)
        Managed Entity Instance: 0x0101
    Result: Unknown managed entity (04)
    Trailer
        CPCS-UU and CPI: 0x0000
        CPCS-SDU Length: 0x0028
        CRC32: 0xd37cb2ce

Frame 790: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: Send_00 (20:53:45:4e:44:00), Dst: Receive_00 (20:52:45:43:56:00)
OMCI Protocol, ONU< Create               - Reserved for future standardization (351)
    Transaction Correlation ID: 40898
    Message Type = Create
        0... .... = Destination Bit: 0x0
        .0.. .... = Acknowledge Request: 0x0
        ..1. .... = Acknowledgement: 0x1
        ...0 0100 = Message Type: Create (4)
    Device Identifier: 0x0a
    Message Identifier, ME Class = Reserved for future standardization (351), Instance = 257
        Managed Entity Class: Reserved for future standardization (351) (015f)
        Managed Entity Instance: 0x0101
    Result: Unknown managed entity (04)
    Trailer
        CPCS-UU and CPI: 0x0000
        CPCS-SDU Length: 0x0028
        CRC32: 0xf685d6fa

martin@lab-svr:~$

So there were four occurrences where ONU signaled a failure back to OLT. All four are related to managed entity ID 351. Example of a message sent by the OLT triggering the Unknown managed entity response is following:
OMCI Protocol, OLT> Create               - Reserved for future standardization (351)
    Transaction Correlation ID: 38560
    Message Type = Create
        0... .... = Destination Bit: 0x0
        .1.. .... = Acknowledge Request: 0x1
        ..0. .... = Acknowledgement: 0x0
        ...0 0100 = Message Type: Create (4)
    Device Identifier: 0x0a
    Message Identifier, ME Class = Reserved for future standardization (351), Instance = 257
        Managed Entity Class: Reserved for future standardization (351) (015f)
        Managed Entity Instance: 0x0101
    Attribute List
    Trailer
        CPCS-UU and CPI: 0x0000
        CPCS-SDU Length: 0x0028
        CRC32: 0xa3e8db2f

However, I doubt that this reserved managed entity class has something to do with Extended VLAN tagging operation configuration data.

have you change the mac address of the stick?

have you change the mac address of the stick?

Do you mean the ELAN_MAC_ADDR setting? If yes, then I did try to change this. I tried with both the lan mac and wan mac seen in the output of Huawei HG8010H display productmac command. MAC_KEY was also updated accordingly.
As a workaround at this stage, I'm running a systemd service on my Linux-based router which runs a script adjusting the forward operation if needed:
#!/usr/bin/env bash

exec 2>/dev/null

while true; do

    filter_data=$(timeout 5 sshpass -p admin ssh<EMAIL_ADDRESS>"omcicli mib get 84 0x01")

    if grep -qs "^FwdOp: *0x04$" <<< "$filter_data"; then
        timeout 5 sshpass -p admin ssh<EMAIL_ADDRESS>"omcicli mib set 84 0x01 FwdOp 0x02"
    fi

    sleep 10

done

This restores the connectivity, for example, after the SFP ONT reboot.

Excuse me @ToTheCLI, where did you get the firmware M110_sfp_ODI_240202.tar? I can't find anywhere to download the latest versions, maybe an official site.
Is the version M110_sfp_ODI_240202.tar hybrid? which seem to be the ones recommended at https://hack-gpon.org/ont-odi-realtek-dfp-34x-2c2/ Is this indicated by the nomenclature M110_* or M114_*?

Excuse me @ToTheCLI, where did you get the firmware M110_sfp_ODI_240202.tar? I can't find anywhere to download the latest versions, maybe an official site.

In the Discord thread for the realtek chipset

Try flash set OMCC_VER 134

@rasMUX Thanks! In case of flash set OMCC_VER 134 I indeed receive the Extended VLAN tagging operation configuration data. However, the ONT stays authenticated(O5 state) only for few seconds:
# diag gpon get onu-state
gpon get onu-state
ONU state: Standby State(O2)
# diag gpon get onu-state
gpon get onu-state
ONU state: Serial Number State(O3)
# diag gpon get onu-state
gpon get onu-state
ONU state: Ranging State(O4)
# diag gpon get onu-state
gpon get onu-state
ONU state: Operation State(O5)
RTK.0> command:#
# diag gpon get onu-state
gpon get onu-state
ONU state: Operation State(O5)
# diag gpon get onu-state
gpon get onu-state
ONU state: Operation State(O5)
# diag gpon get onu-state
gpon get onu-state
ONU state: Standby State(O2)
RTK.0> command:#
#

Cycle above seems to repeat forever. diag gpon get onu-state commands were executed roughly a second apart.
The configuration I'm using can be seen below:
flash set GPON_SN HWTC00112233
flash set PON_VENDOR_ID HWTC

flash set OMCI_OLT_MODE 3
flash set GPON_ONU_MODEL HG8010H
flash set HW_HWVER 2B2.A
flash set OMCI_SW_VER1 V1.7.1
flash set OMCI_SW_VER2 V1.7.1
flash set OMCC_VER 134

flash set OMCI_FAKE_OK 1

flash set ELAN_MAC_ADDR 80d4a5112233
flash set MAC_KEY 400c594b278d89035dc3db37bbc45d0e

flash set HW_CWMP_MANUFACTURER 'Huawei Technologies Co., Ltd'
flash set HW_CWMP_PRODUCTCLASS SFU

The values of GPON_SN and ELAN_MAC_ADDR were obfuscated. In case I change the OMCC_VER back to its default value of 128, then the ONT authenticates just fine.
In addition, I have tried with various other OMCC_VER values like 136 or 160 but the authentication problem persists.

can someone please provide me firmware M110_sfp_ODI_240202.tar

can someone please provide me firmware M110_sfp_ODI_240202.tar

M110_sfp_ODI_240202_Extract_Tar_File.zip
Extract the .tar File from the archive

@tonusoo did it work for you?

@myepezg Do you mean the firmware upgrade to V1.1.6-240202? If yes, then this did not help.

@tonusoo Ok ok. And did any method work for you?

Download Link broken

the firmware upgrade work for me


Binary organization

With the addition of siae (and soon siam?), I think we need to have a discussion on how these tools should fit into the Sia project.
Our current approach is to implement each new tool as a module, add its endpoints to the API, and then build a new binary in the style of siad that initializes an API server with the desired set of modules. To this end, the API server now checks for nil modules so that it only serves requests for non-nil modules.
An alternative would be to package everything into one binary, and then use build tags to select which modules would be included (e.g. make release modules=consensus,gateway,explorer for siae). This breaks when you want siae to use different flags from siam, though.
Another option would be to separate these tools into their own repo(s) instead of implementing them as modules. After all, siae and siam are not used by the other modules. This would mean reimplementing the API server (albeit in much simpler fashion) for these tools. A nice result of this would be the removal of the nil checks in the current API server, and a cleaner separation of these tools from the primary codebase. This is the option I'm leaning towards.

I'm partial to build flags, as it would keep the code centralized for future changes, but could also produce binaries dedicated to one function.
To put another alternative out there, we could use runtime flags to determine which modules to load (e.g. siad +explorer -wallet -renter -miner -host ...) with sane defaults for ordinary use.

I don't see why we can't just have runtime flags.
siad by default would have --renter, and then optionally you could enable --miner --host --explorer --minerpool or you could disable --renter. 'siac' I think makes sense as a separate binary - it's completely different from everything else.
Each of those will automatically pull in all the dependencies needed. Renter has wallet + hostdb as dependencies, and those two would pull in things like consensus and gateway.
I'm also in favor of ditching cobra if possible, as well as gcfg (which I don't think we use at all anyway). 'siac' is okay to keep cobra, the use-case seems very strong.

Hmm. I'm waffling on this one. I think it really depends on what sort of functionality the block explorer and the mining pool will have. I worry that we are succumbing to feature bloat and straying from "do one thing well."

It makes sense for everything to be in one larger binary because the amount of shared code is extraordinary.

Not really. The majority of what's shared can be easily imported. If you moved siae to its own repo, it would just be the files in /modules/blockexplorer and /siae. You wouldn't need to copy over consensus, gateway, etc.

true, but you're still doing things like subscriptions, which is unique to the 'siad' group of modules.
All that the binaries do is grant access to the api of the module system of siad.
I am wary of fragmenting.

Siae and siad have been merged.


Tiger Tree Hash - API / Code Mismatch

Header and code disagree on the size of the resulting hash/digest.
tth.h:23: void rhash_tth_final(tth_ctx* ctx, unsigned char result[64]);
tth.c:93: void rhash_tth_final(tth_ctx* ctx, unsigned char result[24])
..................................................................^

The header claims a digest size of 64 bytes, the implementation uses 24 only.
Given that TTH is a hash tree based on Tiger, and Tiger's hash size is 24, I suspect that the typo is in the header/declaration.
Maybe define a tth_hash_length (or tth_has_size) macro ?

Thanks for reporting this issue! It's fixed by 821b030830771dcce4b660cd1cfc463d8374f2fa.

You are welcome. (Do I (have to) close the ticket now ? What with the done label and still open).
FYI, the issue was originally found when testing https://github.com/andreas-kupries/ankh and the hash results were unstable in the lower bytes (i.e. random memory data).


Arity of constructors not checked when pattern matching

The following code is bogus, Leaf does not take any arguments:
type tree =
  | Leaf : tree

val in_tree : int -> tree -> Tot bool
let rec in_tree x t =
  match t with
  | Leaf n -> false (* Note: Leaf doesn't take arguments! *)

Still this passes type-checking:
time mono ../../bin/fstar.exe --fstar_home ../.. bug156.fst
Verified module: Prims
Verified module: Bug
All verification conditions discharged successfully
mono ../../bin/fstar.exe --fstar_home ../.. bug156.fst  1.66s user 0.02s system 101% cpu 1.661 total

What?! Very surprised.


<EMAIL_ADDRESS>deprecated due to ReDOS vulnerability

Received this npm warning today installing Elm.
npm WARN deprecated<EMAIL_ADDRESS>ReDoS vulnerability parsing Set-Cookie

Reference: https://nodesecurity.io/advisories/130

Elm is using an old version. Moving this issue to Elm.

link to the elm issue: https://github.com/elm-lang/elm-platform/issues/166


Add Docker based readme docs

Some docs based on https://github.com/ethereumjs/testrpc/commit/649f79f7ede14f8ee718d47ff35e4e8423d28afe just merged.
assumes the dockerhub repo gets set up at ethereumjs/testrpc...

Someone should create that account! Currently (as of 20140408.0121EST) is a 404

Can this or #67 be merged?

Merged by hand. Added as a larger section at the bottom of the README.


Base62 Implementations that do and don't agree

It seems that the algorithms associated with "base_x", GMP, GnuPG, and Bitcoin Base58 all seem to agree:
(at least in simple tests of a few dozen bytes or less)

https://github.com/oconnor663/basex_gmp/issues/1
https://github.com/seruco/base62/issues/4
https://github.com/brianhempel/base_x/issues/1
https://en.wikipedia.org/wiki/Base62

Disagree

https://github.com/lytics/base62/issues/6, https://github.com/lytics/base62/issues/3
https://github.com/jxskiss/base62/issues/2
https://github.com/abhishekjhaji/go-base62/issues/1

Unknown

https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
https://gmplib.org/
https://github.com/eknkc/basex
https://codeberg.org/ac/base62
https://github.com/marksalpeter/token
https://github.com/tuupola/base62/issues/29

Reference Base62 Strings
Raw   : Hello, 世界 (13 bytes)
Base64: SGVsbG8sIOS4lueVjA (18 chars)
Base62: 1wJfrzvdbuFbL65vcS (18 chars)

Raw   : Hello World (11 bytes)
Base64: SGVsbG8gV29ybGQ (15 chars)
Base62: 73XpUgyMwkGr29M (15 chars)

Raw   : [0] (1 bytes)
Base64: AA (2 chars)
Base62: 00 (2 chars)

Raw   : [0 0 0 0 0 0 0 0 0 0 0 0] (12 bytes)
Base64: AAAAAAAAAAAAAAAA (16 chars)
Base62:<PHONE_NUMBER>0000000 (17 chars)

Raw   : [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] (24 bytes)
Base64: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA (32 chars)
Base62:<PHONE_NUMBER>00000000000000000000000 (33 chars)

Raw   : [0 0 0 0 255 255 255 255] (8 bytes)
Base64: AAAAAP____8 (11 chars)
Base62: 000004gfFC3 (11 chars)

Raw   : [255 255 255 255 0 0 0 0] (8 bytes)
Base64: _____wAAAAA (11 chars)
Base62: LygHZwPV2MC (11 chars)

Test Code Reference
package main

import (
	"encoding/base64"
	"fmt"

	"github.com/keybase/saltpack/encoding/basex"
)

func main() {
	for _, src := range [][]byte{
		[]byte("Hello, 世界"),
		[]byte("Hello World"),
		{0},
		{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
		{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
			0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0},
		{0, 0, 0, 0, 255, 255, 255, 255},
		{255, 255, 255, 255, 0, 0, 0, 0},
	} {
		b62 := basex.Base62StdEncoding.EncodeToString(src)
		b64 := base64.RawURLEncoding.EncodeToString(src)

		if src[0] == 0x0 || src[1] == 255 {
			fmt.Printf("Raw   : %v (%d bytes)\n", src, len(src))
		} else {
			fmt.Printf("Raw   : %v (%d bytes)\n", string(src), len(src))
		}
		fmt.Printf("Base64: %s (%d chars)\n", b64, len(b64))
		fmt.Printf("Base62: %s (%d chars)\n", b62, len(b62))
		fmt.Println("")
	}
}


Improve Instructions related to how-to configure IBM FHIR Server on DockerHub

Is your feature request related to a problem? Please describe.
Improve Instructions related to how-to configure IBM FHIR Server on DockerHub
Describe the solution you'd like
Improve the inline documentation to describe layer or volume configuration or externalized parameters
Describe alternatives you've considered
n/a
Additional context
n/a

Draft posted to https://github.com/IBM/FHIR/wiki/Docker-Hub---Draft
@ldettman @lmsurpre  please review when you are available - no rush.
### Copy the configuration as a Layer

Create your own Dockerfile and add a layer to copy your configuration to the image

FROM ibmcom/ibm-fhir-server:your-version
COPY --chown=1001:0 fhir-server-config.json /opt/ol/wlp/usr/servers/fhir-server/config/default
COPY --chown=1001:0 overrides/ /opt/ol/wlp/usr/servers/fhir-server/configDropins/

### Put the Configuration into a Volume
Copy the existing [configurations](https://github.com/IBM/FHIR/tree/master/fhir-server/liberty-config/config/default) files to a local folder `<local_fhir_config_dir>`

Create a folder for overrides on your local system `<local_liberty_overrides_dir>`

Run your Docker with the local folders mounted as volumes

docker run -p9443:9443 ibmcom/ibm-fhir-server -v <local_fhir_config_dir>:/opt/ol/wlp/usr/servers/fhir-server/config/
-v <local_liberty_overrides_dir>:/opt/ol/wlp/usr/servers/fhir-server/configDropins/overrides

Note: other combinations and volume mounts are possible. The power with this technique is your configuration is not on ephemeral storage.

Next action: post review, update docker hub https://hub.docker.com/repository/docker/ibmcom/ibm-fhir-server

Reviewed and looks good.

Looks good to me too.
Maybe insert one word on this line to make it clear its for server.xml overrides and not fhir-server-config overrides?
From

Create a folder for overrides on your local system <local_liberty_overrides_dir>
to
Create a folder for server.xml overrides on your local system <local_liberty_overrides_dir>

Done, going to make the edit and update


chore: clean npm cache on package installations

This should avoid persistent errors in case of rare npm cache corruptions.
Something like https://github.com/immerjs/immer/issues/546.

@nolanmar511 I just noticed this - thanks for fixing the PR title. I always forget using this semantic versioning compatible style.


runtime error when restart button pressed

Pressing the restart button immediately results in this JS error:

(Uncaught TypeError: Cannot read property 'time' of undefined

Using Elm 0.15.1

Happening because we're trying to get the 0th event in runtime.timer.now, but there are no events because we just cleared them all. Need to get the time from some other source.

Seems like a job for a weird special case if.

Yup!

I'm trying out elm for the first time and I'm also running into this issue. Less than ideal developer experience. +1 to fix this

Having the same issue. Am I doing something wrong?

also get this error, how to fix it ?

This makes me sad 😕

+1

+1

Tried elm for the first time today, using the Hello World examples project. The first thing I see are errors like this.

+1


Cleanup dependencies in taker-frontend

Read the commit messages for details.

nice one! really nice that it's also covered with the lint :)

Anything that doesn't make the build fail will come back ...

bors r+


Git tag for version 21.0.0 is missing

I see there is https://mvnrepository.com/artifact/io.foojay.api/discoclient/21.0.0, which is never than release 2.0.39. So, has the versioning scheme changed?

I'm starting to believe that not the Git tag is missing, but that the library version 21.0.0 was published by accident. Could that be @HanSolo?

Just checked that and both is true, the tag is missing and the version was published by accident. The reason is that the discoclient is nothing that I officially support, but was done to help the NetBeans team to integrate the Disco API into their IDE. The new version scheme follows the JDK.MAJOR.MINOR versioning. Meaning to say the 21.0.0 is based on JDK 21 and I see in my local repo that I already have a 21.0.1 version which is not published yet. It doesn't contain any major changes but more or less cosmetics

The reason is that the discoclient is nothing that I officially support

As a related question, is there a supported API client for the Disco API (in Java or Kotlin)? Or is the recommendation to use the API directly (like I believe https://github.com/gradle/foojay-toolchains does), e.g. via Retrofit?

The DiscoClient does it's job, so feel free to use it but as mentioned it was made with a specific purpose in mind, you can easily directly access the api and get the data you need.


docs: code miss in react three fiber tutorial

Hi, Thank you for the react three fiber tutorial.
I found one mistake on loading textures page.
https://docs.pmnd.rs/react-three-fiber/getting-started/loading-textures
incorrect: <sphereGeometry args={[1, 100, 100]} map={colorMap} />
correct: <meshStandardMaterial  map={colorMap} />

Thanks for investigating. Fixed with #135.


[WFLY-14074] Normalize principial propagation/injection across elytron…

… and legacy
https://issues.redhat.com/browse/WFLY-14074

wildfly-ejb3 schema was already at version 8. So you will need to create a new schema file, like wildfly-ejb3_9_0.xsd, to contain your additions.  If there is other features later within the current release that will add to the schema, they can modify on top of your version of ejb3 schema.

@fjuma @chengfang I added another commit with update of EJB3 schema from 8.0 to 9.0. Thanks!

I just added community documentation to this PR.

@chengfang Please review.

wildfly-ejb3 schema was already at version 8. So you will need to create a new schema file, like wildfly-ejb3_9_0.xsd, to contain your additions. If there is other features later within the current release that will add to the schema, they can modify on top of your version of ejb3 schema.

@chengfang After I updated ejb3 schema to 9.0 I started to get mixed domain failures and I am not yet able to figure out what is wrong. Do you maybe have any idea what could be wrong with my schema update commit? Thank you

wildfly-ejb3 schema was already at version 8. So you will need to create a new schema file, like wildfly-ejb3_9_0.xsd, to contain your additions. If there is other features later within the current release that will add to the schema, they can modify on top of your version of ejb3 schema.

@chengfang After I updated ejb3 schema to 9.0 I started to get mixed domain failures and I am not yet able to figure out what is wrong. Do you maybe have any idea what could be wrong with my schema update commit? Thank you

@Skyllarr are you still seeing mixed domain failures? It could be the new attribute is not properly handled by EJBTransformers. Which tests failed?

wildfly-ejb3 schema was already at version 8. So you will need to create a new schema file, like wildfly-ejb3_9_0.xsd, to contain your additions. If there is other features later within the current release that will add to the schema, they can modify on top of your version of ejb3 schema.

@chengfang After I updated ejb3 schema to 9.0 I started to get mixed domain failures and I am not yet able to figure out what is wrong. Do you maybe have any idea what could be wrong with my schema update commit? Thank you

@Skyllarr are you still seeing mixed domain failures? It could be the new attribute is not properly handled by EJBTransformers. Which tests failed?

The following classes have failing tests:
org.jboss.as.test.integration.domain.mixed.eap720.MixedDomain720TestSuite
org.jboss.as.test.integration.domain.mixed.eap720.MixedDomainOverlay720TestSuite
org.jboss.as.test.integration.domain.mixed.eap730.MixedDomain730TestSuite
org.jboss.as.test.integration.domain.mixed.eap730.MixedDomainOverlay730TestSuite

error is mostly "WFLYDC0022: Operation failed or was rolled back on all servers." It might be because of EJBTransformers, however the test Ejb3TransformersTestCase is passing and I can't figure out what is wrong with the existing transformers.

I'm also seeing the mixed-domain failure in my local testing. I'll take a closer look. In the meantimem, you may want to ask in Zulip chat about "WFLYDC0022: Operation failed or was rolled back on all servers."

I'm also seeing the mixed-domain failure in my local testing. I'll take a closer look. In the meantimem, you may want to ask in Zulip chat about "WFLYDC0022: Operation failed or was rolled back on all servers."

@chengfang  Thank you a lot, I will investigate further as well. And just FYI, only the single commit titled Update EJB3 schema from 8.0 to 9.0 from this PR is sufficient to make these tests fail.

@chengfang The logs of the slave server-one have the following error:
ERROR [org.jboss.as.server] (ServerService Thread Pool -- 76) WFLYSRV0021: Deploy of deployment "test.war" was rolled back with the following failure message: undefined

and
WFLYCTL0184:    New missing/unsatisfied dependencies:
      service jboss.remoting.remotingConnectorInfoService.undefined (missing) dependents: [service org.wildfly.clustering.cache.registry-entry.ejb.client-mappings]
WFLYCTL0448: 4 additional services are down due to their dependencies being missing or failed

Unfortunately I still can't find anything in my schema update that is making this error happen.

Ejb3TransformersTestCase is missing transformer tests for EAP 7.3.

Thank you for review! I added this test case, hopefully the versions there are correct.

@pferraro Thank you for review! I added this test case, hopefully the versions there are correct.

@pferraro Thank you for review! I added this test case, hopefully the versions there are correct.

We also need an EAP 7.3 version of the rejection style transformer test, i.e. testRejectionsEAP730()

@pferraro I added this test. Thank you.

@OndrejKotek let me know whether pre-check can be granted to this PR. Thanks!

@OndrejKotek let me know whether pre-check can be granted to this PR. Thanks!

I have executed WildFly and EJB test suites on our CI, no regressions found, pre-checked.

@OndrejKotek  @chengfang Thank you both.

@fjuma @pferraro Please complete your review so we can retest this one and get it in. Thanks!

/retest


Please add a --stdout and --stderr to the tap runner

It's really useful to be able to get my console.log()'s to show up when writing tests, until stuff is in a "steady state".

you can kind of get around this by exporting an environment var on your shell:
export TAP_DIAG=1
and you'll get the truncated console output as well as some other diagnostic info.

also, another BS workaround is to:
throw util.inspect(someObject)
not remotely a good solution but is currently getting me the object inspection I need

Setting TAP_DIAG works, but it of course causes much more output than is desired.
The inability to use console.log in test code and in the code being tested is a serious issue for debugging!

+1

+!

I've added a --debug option which pipes stderr through from your tests.
+1 these two pull requests:
https://github.com/isaacs/tap-runner/pull/4
https://github.com/isaacs/node-tap/pull/15

Should this even require an option? I think that stderr should probably just be emitted as is.
Alternately, for debugging, it'd be nice to have a test.diag() function which outputted TAP diagnostics (http://testanything.org/wiki/index.php/TAP_specification#Diagnostics)

I also have a pull request which forwards stderr all the time and includes a test to verify.
https://github.com/isaacs/node-tap/pull/31
FYI. Also for any of those who need this asap, you can use my tap runner (tapr) which provides stderr and stdout, optional colorized output, and a cleaner assert formatting. It was based on the original tap runner and still uses all other original tap components just with a customized runner. There are some screenshots on my github page https://github.com/jeffbski/tapper and it is available in npm as tapr


TLS migration for demos/web

What does this PR do?
Minimally invasive migration to TLS for the http (8000) and websocket (9000) endpoints for the real time web demo.
Where should the reviewer start?
Install the demo like a normal install. During the install-deps.sh script it will prompt for questions to generate a local self-signed cert. Anything can be entered into the cert. Start the demo as normal but connect on https://domain:http_port instead of http.
How should this PR be tested?
The first step is the cert generation (added to install-deps.sh for convenience). The second step is ensuring the two endpoints are brought up. The next step is loading the web page and accepting the self-signed cert. Reload the web app and everything should now be secure with no errors or warnings and the video works as normal.
Any background context you want to provide?
I tried to be as minimal as possible so the changes tends to follow the existing structure rather than a refactor of anything major. With that said, SimpleHTTPServer doesn't work with TLS via the -m flag so that is now a short script. I updated all the html/js files to point to https/wss. I also updated the js for Firefox's change from navigator.mozGetUserMedia to navigator.mediaDevices.getUserMedia.
This was tested on Ubuntu 14.04
What are the relevant issues?
#75
Screenshots (if appropriate)
Questions:

Do the docs need to be updated?

Yes. I updated the script's docs in demos/web but didn't make any changes outside of demos/web

Does this PR add new (Python) dependencies?

No but some machines might need openssl if it's not already installed.

Great PR! Thanks.


CUDA 10

Edited: 17-DEC-2018 Nightly builds are CUDA 10
Edited: 27-NOV-2018 Added 1.12 FINAL builds.
Edited: 29-OCT-2018 Added 1.12 RC2 builds.
Edited: 17-OCT-2018 Added 1.12 RC1 builds.
Nightly Builds are now CUDA 10 as of 16-DEC-2018

[DONE] CUDA 10 in the nightly builds mid-DEC 2018 if testing goes well
CUDA 10 in official in mid-JAN-2019 as TF 1.13.

Install instructions for CUDA 10 for a fresh system that should also work on an existing system if using apt-get.
TensorFlow will be upgrading to CUDA 10 as soon as possible.  As of TF 1.11 building TensorFlow from source works just fine with CUDA 10 and possibly even before.  There is nothing special needed other than all of the CUDA, cuDNN, NCCL (optional), and TensorRT (optional) libraries.  If people have some builds feel free to link them here (Keep in mind if you download them to decided what risk you want to take based on the source.) as well as any issues.  Also NCCL is now open source again and soon will be back to being automatically downloaded by bazel and included in the binary.
CUDA 10 would likely go into TF 1.13, which is not scheduled.  I will update this post as I have more info to share.  I hope to flip nightly builds to CUDA 10 in November but the TF 1.13 release will likely push to early Jan.
I and some really cool people below made some binaries (even windows, see comments below) to help people out along with my really bad "instructions".  I suspect the instructions linked in the comments below are better.
Libraries used (rough list, similar to what I listed above)

Ubuntu 16.04 LTS on GCE from base Google Cloud Ubuntu image.
Python 2.7 because that is how I roll and someday I will change.
Install 410+ driver using apt-get
CUDA 10.0.130 from .tgz install (do not install the driver)
cuDNN-10.0 v<IP_ADDRESS> rom tgz install
nccl 2.3.4 for CUDA 10 (I am not sure it matters I have mixed them up before)
TensorRT-5.0.0 for CUDA 10 cudnn 7.3 (TensorRT 5 was not stated as supported by Tensorflow when
Compute Capability: 3.7,5.2,6.0,6.1,7.0

TF 1.12.0 FINAL
Python 2.7 (Ubuntu 16.04)

AVX2 + CUDA 10

Python 3.5 (Ubuntu 16.04)

AVX2 + CUDA 10

Dockerfiles
These are partial files and the apt-get commands should work on non-Docker systems as well assuming you have the NVIDIA apt-get repositories which should be the same as listed on tf.org.

Devel  Confirmed by building TensorFlow at 1.12RC0.
Runtime

I've built tf with all the latest, test well except this silly warning
*** WARNING *** You are using ptxas 10.0.145, which is older than 9.2.88. ptxas 9.x before 9.2.88 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
You do not need to update to CUDA 9.2.88; cherry-picking the ptxas binary is sufficient.
seems no problem for now

Could we expect CUDA10+python3.7 prebuilt images from 1.13rc0?

@alanpurple   Ahh I had not tried CUDA 10 with XLA yet, it is now compiled in by default.  Great feedback I will pass that on to the team.  As far as Python versions that is not under my authority (even a little).  @gunan owns that part of the matrix.

@tfboyd Hey could you kindly share the .whl of tensorflow 1.11 with,
CUDA 10.0.130_410.48
cuDNN <IP_ADDRESS> for CUDA 10
I was not able to build the same. I have a rtx 2080 with ubuntu 16.04. Thanks in advance

@tejavoo Let me try to make one or a few for 16.04 as one offs.  The ones I had ended during a period of time where we had a session memory leak that did not impact my tests but I feel bad sharing them.  I will see what I can do.

Just wrote instruction how to build TF from scratch. Maybe somebody found it useful

Here is my new tutorial for Building Tensorflow 1.12 + CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5 + bazel-0.17.2 https://www.python36.com/how-to-install-tensorflow-gpu-with-cuda-10-0-for-python-on-ubuntu/, you will also get prebuilt wheel at last with AVX and compute capability 5.0.

Thanks @arunmandal53 for taking the time to write the tutorial. However, I need to install Tensorflow in the Python installation that is under my home account rather than  installing Tensorflow on Python that comes with Ubuntu. I think that I need to wait until Tensorflow releases a version that can be installed using pip and is compatible with CUDA 10 (and cuDNN)

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5

@ivan-marroquin  Incase you did not know you can install any .whl file anywhere you want.  You can use the .whl packages I created or the ones @arunmandal53  has done which covers Python 3.6.  Pretty nice selectoin :-).  I might also not understand you issue but you can do this:
pip install http://path to whatever you want
or
pip install /local/path/to/whl
You do not have to count on pypi and --user would also work and installing in a virtual env.

@tfboyd thanks for the clarification. And thanks to all of you for a such impressive help! i will give a try as soon as possible

@everyone thanks for spilling all the bits and bytes. I have built the TF with all my requirements and successfully running the tests for 5days now :)

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + python 3.6 + Windows

Here is my new tutorial  on building Tensorflow 1.12 + CUDA 10.0 + CUDNN 7.3.1 + + Bazel 0.17.2 https://www.python36.com/how-to-install-tensorflow-gpu-with-cuda-10-0-for-python-on-windows/ .

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5 + python 3.6 +Ubuntu

do you have tensorflow whl for windows 10?

You wil find whl link above.

Detailed tutorial just in case :)
https://medium.com/@saitejadommeti/building-tensorflow-gpu-from-source-for-rtx-2080-96fed102fcca

Ubuntu 18.04 - CUDA 10.0 - libcudnn 7.3.1 + Python 3.6
I have tried the various wheel packages but it coredumps. :(

Here is a windows build tensorflow 1.12, CUDA 10, cuDNN 7.3.1, RTX2080TI (feature level 7.5).
tensorflow-1.12.0-cp36-cp36m-win_amd64.whl

are nightlies cuda 10 at the moment?

Here is a ubuntu 18.109 build, tensorflow 1.12 release branch, python 3.6, CUDA 10.0, cuDNN <IP_ADDRESS>, NCCL 2.3.7, tensorRT 5.0, compute capability 7.5, 6.1, for my RTX 2080 TI.
tensorflow-1.12.0-cp36-cp36m-linux_x86_64.whl

@lahwran  We are in the process of making the nightly builds CUDA 10.  Some may have flipped already.  I will update over the next few days in this comment and the original post at the top.

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5 + python 3.6 +Ubuntu

was of great help and worked

Hi,
Could you publish the compile options you used? I tried to compile the tensorflow myself, but found that the training throughput is slower that what you provide? I'm trying to figure out what's the difference?
Thanks

Took my 15 days to followup, sorry.  This is the timeline I am seeing right now:

CUDA 10 in the nightly builds mid-DEC if testing goes well
CUDA 10 in official in mid-JAN due to the holiday break I would guess TF 1.13 (1.12 is out already)

I am adding this to the original post above as well.

Took my 15 days to followup, sorry. This is the timeline I am seeing right now:

CUDA 10 in the nightly builds mid-DEC if testing goes well
CUDA 10 in official in mid-JAN due to the holiday break I would guess TF 1.13 (1.12 is out already)

I am adding this to the original post above as well.

Hope everything goes well and TF 1.13 appear as soon as possible. Because of the version issue, I can't install TensoFlow on my computer.

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5 + python 3.6 +Ubuntu

I tried to install the .whl you provided, but encountered a problem like this:
Ignoring visible gpu device (device: 0, name: GeForce GT 730, pci bus id: 0000:01:00.0, compute capability: 3.5) with Cuda compute capability 3.5. The minimum required Cuda capability is 5.0.
I admit my GPU is too old, but is there any method to solve this problem?

Here is a ubuntu 18.10 build (kernal 4.18), tensorflow r1.12 branch, python 3.6, CUDA 10.0, cuDNN <IP_ADDRESS>, NCCL 2.3.7, tensorRT 5.0, compute capability 7.5, 6.1, for my RTX 2080 TI.
tensorflow-1.12.0-cp36-cp36m-linux_x86_64.whl

@MingyaoLiu Were you able to install CUDA 10 on Ubuntu 18.10 ? Did you use 18.04 deb package?

Here is a ubuntu 18.10 build (kernal 4.18), tensorflow r1.12 branch, python 3.6, CUDA 10.0, cuDNN <IP_ADDRESS>, NCCL 2.3.7, tensorRT 5.0, compute capability 7.5, 6.1, for my RTX 2080 TI.
tensorflow-1.12.0-cp36-cp36m-linux_x86_64.whl

@MingyaoLiu Were you able to install CUDA 10 on Ubuntu 18.10 ? Did you use 18.04 deb package?

Yes. If i remember correctly i manually installed with cuda 10 for 18.04 run file. Deb may not work?

Here is a ubuntu 18.10 build (kernal 4.18), tensorflow r1.12 branch, python 3.6, CUDA 10.0, cuDNN <IP_ADDRESS>, NCCL 2.3.7, tensorRT 5.0, compute capability 7.5, 6.1, for my RTX 2080 TI.
tensorflow-1.12.0-cp36-cp36m-linux_x86_64.whl

@MingyaoLiu Were you able to install CUDA 10 on Ubuntu 18.10 ? Did you use 18.04 deb package?

Yes. If i remember correctly i manually installed with cuda 10 for 18.04 run file. Deb may not work?

Hi @MingyaoLiu ,
Can you share the .whl file you compiled?
Because my GPU's compute capability is 3.5 . But the .whl file provided by @arunmandal53 requires 5.0, so it doesn't work on my computer.

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + python 3.6 + Windows

What compute level?
Can you please share how you make the file? When I tried I did not succeed :(

Here is a windows build tensorflow 1.12.0, CUDA 10, cuDNN 7.3.1, RTX2080TI (feature level 7.5).
tensorflow-1.12.0-cp36-cp36m-win_amd64.whl
one above in the comment I'm not sure what the CUDA feature level is, so I build a new one.

Can you please share how you make the file? When I tried I did not succeed :(
I need the file with compute level 7...

When I try to build TF at AWS P3 V100 Tesla instance it runs 4000/6000 and then fails? What can I do to fix it?
.\tensorflow/core/kernels/mirror_pad_op_cpu_impl.h(31): note: see reference to class template instantiation 'tensorflow:
:functor::MirrorPadtensorflow::CpuDevice,tensorflow::int64,tensorflow::int64,5' being compiled
ERROR: C:/tensorflow/tensorflow/core/kernels/BUILD:3568:1: undeclared inclusion(s) in rule '//tensorflow/core/kernels:so
ftsign_op_gpu':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/kernels/softsign_op_gp
u.cu.cc':
'C:/users/administrator/appdata/local/temp/nvcc_inter_files_tmp_dir/softsign_op_gpu.cu.cudafe1.stub.c'
'C:/users/administrator/appdata/local/temp/nvcc_inter_files_tmp_dir/softsign_op_gpu.cu.fatbin.c'
c:\users\administrator_bazel_administrator\xv6zejqw\execroot\org_tensorflow\external\eigen_archive\eigen\src/Core/util/
Memory.h(164): warning: calling a host function from a host device function is not allowed
later it ends with...
c:\users\administrator_bazel_administrator\xv6zejqw\execroot\org_tensorflow\external\eigen_archive\unsupported\eigen\sr
c/SpecialFunctions/SpecialFunctionsImpl.h(712): warning: missing return statement at end of non-void function "Eigen::in
ternal::igamma_series_impl<Scalar, mode>::run [with Scalar=double, mode=Eigen::internal::SAMPLE_DERIVATIVE]"
detected during:
instantiation of "Scalar Eigen::internal::igamma_series_impl<Scalar, mode>::run(Scalar, Scalar) [with Scalar
=double, mode=Eigen::internal::SAMPLE_DERIVATIVE]"
(863): here
instantiation of "Scalar Eigen::internal::igamma_generic_impl<Scalar, mode>::run(Scalar, Scalar) [with Scala
r=double, mode=Eigen::internal::SAMPLE_DERIVATIVE]"
(2108): here
instantiation of "Eigen::internal::gamma_sample_der_alpha_retval<Eigen::internal::global_math_functions_filt
ering_base<Scalar, void>::type>::type Eigen::numext::gamma_sample_der_alpha(const Scalar &, const Scalar &) [with Scalar
=double]"
c:\users\administrator_bazel_administrator\xv6zejqw\execroot\org_tensorflow\external\eigen_archive\unsupported\eigen\sr
c/SpecialFunctions/arch/CUDA/CudaSpecialFunctions.h(154): here
c:\users\administrator_bazel_administrator\xv6zejqw\execroot\org_tensorflow\external\eigen_archive\eigen\src/Core/Array
Wrapper.h(94): warning: __declspec attributes ignored
external/com_google_absl\absl/strings/string_view.h(496): warning: expression has no effect
external/protobuf_archive/src\google/protobuf/arena_impl.h(55): warning: integer conversion resulted in a change of sign
external/protobuf_archive/src\google/protobuf/arena_impl.h(309): warning: integer conversion resulted in a change of sig
n
external/protobuf_archive/src\google/protobuf/arena_impl.h(310): warning: integer conversion resulted in a change of sig
n
external/protobuf_archive/src\google/protobuf/map.h(1025): warning: invalid friend declaration
host_defines.h is an internal header file and must not be used directly.  This file will be removed in a future CUDA rel
ease.  Please use cuda_runtime_api.h or cuda_runtime.h instead.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1100.301s, Critical Path: 135.78s
INFO: 2335 processes: 2335 local.
FAILED: Build did NOT complete successfully
PS C:\tensorflow>

Edited: 27-NOV-2018 Added 1.12 FINAL builds.
Edited: 29-OCT-2018 Added 1.12 RC2 builds.
Edited: 17-OCT-2018 Added 1.12 RC1 builds.

CUDA 10 in the nightly builds mid-DEC 2018 if testing goes well
CUDA 10 in official in mid-JAN-2019 due to the holiday break and code freezes.  I would guess TF 1.13 (1.12 is out already)

TensorFlow will be upgrading to CUDA 10 as soon as possible. As of TF 1.11 building TensorFlow from source works just fine with CUDA 10 and possibly even before. There is nothing special needed other than all of the CUDA, cuDNN, NCCL (optional), and TensorRT (optional) libraries. If people have some builds feel free to link them here (Keep in mind if you download them to decided what risk you want to take based on the source.) as well as any issues. Also NCCL is now open source again and soon will be back to being automatically downloaded by bazel and included in the binary.
CUDA 10 would likely go into TF 1.13, which is not scheduled. I will update this post as I have more info to share. I hope to flip nightly builds to CUDA 10 in November but the TF 1.13 release will likely push to early Jan.
I and some really cool people below made some binaries (even windows, see comments below) to help people out along with my really bad "instructions". I suspect the instructions linked in the comments below are better.
Libraries used (rough list, similar to what I listed above)

Ubuntu 16.04 LTS on GCE from base Google Cloud Ubuntu image.
Python 2.7 because that is how I roll and someday I will change.
Install 410+ driver using apt-get
CUDA 10.0.130 from .tgz install (do not install the driver)
cuDNN-10.0 v<IP_ADDRESS> rom tgz install
nccl 2.3.4 for CUDA 10 (I am not sure it matters I have mixed them up before)
TensorRT-5.0.0 for CUDA 10 cudnn 7.3 (TensorRT 5 was not stated as supported by Tensorflow when
Compute Capability: 3.7,5.2,6.0,6.1,7.0

TF 1.12.0 FINAL
Python 2.7 (Ubuntu 16.04)

AVX2 + CUDA 10

Python 3.5 (Ubuntu 16.04)

AVX2 + CUDA 10

TF 1.12.0 RC2
Python 2.7 (Ubuntu 16.04)

AVX2 + CUDA 10

Python 3.5 (Ubuntu 16.04)

AVX2 + CUDA 10

TF 1.12.0 RC1
Python 2.7 (Ubuntu 16.04)

AVX2 + CUDA 10

Python 3.5 (Ubuntu 16.04)

AVX2 + CUDA 10

TF 1.12.0 RC0
Python 2.7 (Ubuntu 16.04)

AVX2 + CUDA 10
AVX + CUDA 10

Python 3.5 (Ubuntu16.04)

AVX2 + CUDA 10

Dockerfiles
These are partial files and the apt-get commands should work on non-Docker systems as well assuming you have the NVIDIA apt-get repositories which should be the same as listed on tf.org.

Devel  Confirmed by building TensorFlow at 1.12RC0.
Runtime

Hi Toby, how's your testing going?
Is CUDA 10 supported by nightly builds now?

Should be fixed (partially at least) by 29b8d495c3d3cbd13eed86f542586c1bd1597e46.

@alifare
Nightly builds are now CUDA 10.  I tested via the Docker images from dockerhub.  I did not test Windows as that is something I have yet to use (sorry) for ML. This happened on Friday so there may still be some issues as we roll it out.  Let me know if you run into any problems.  We also moved to NCCL from source so you no longer need to install NCCL to use it.
More info in the new few weeks, I do not know if we will do an official RC before the year ends but 1.13 is likely to go final mid/end-JAN and it 100% has CUDA 10 and cuDNN 7.4 (you could always update this by just adding a new binary).

How can I remove this welcome message cause it crush down the sftp?

@tfboyd Hey could you kindly share the .whl of tensorflow 1.11 with,
CUDA 10.0.130_410.48
cuDNN <IP_ADDRESS> for CUDA 10
I was not able to build the same. I have a rtx 2080 with ubuntu 16.04. Thanks in advance

Me too.
Is there any info?

Tensorflow 1.12 whl for CUDA 10.0 + CUDNN 7.3.1 + NCCL 2.3.5 + python 3.6 +Ubuntu

Is this official?

@alifare
Nightly builds are now CUDA 10. I tested via the Docker images from dockerhub. I did not test Windows as that is something I have yet to use (sorry) for ML. This happened on Friday so there may still be some issues as we roll it out. Let me know if you run into any problems. We also moved to NCCL from source so you no longer need to install NCCL to use it.
More info in the new few weeks, I do not know if we will do an official RC before the year ends but 1.13 is likely to go final mid/end-JAN and it 100% has CUDA 10 and cuDNN 7.4 (you could always update this by just adding a new binary).

Thank a lot @tfboyd
I installed the nightly builds yesterday and I am testing it with sample codes now. Everyting seems to be going well now. I will let you know if I find any probelm in future.

Thank you guys, it also works for me as well with a 1080 ti

Hi @tfboyd , I have the same question as @tydlwav , any updates on the 1.13 release? I am coding in R using Keras and using the RTX 2080. Really looking forward to this release.

rc0 is already out:
https://github.com/tensorflow/tensorflow/releases/tag/v1.13.0-rc0
All our nightlies also have cuda 10 support for a while now.

I've built tf 1.12.0rc0 with all the latest, test well except this silly warning
*** WARNING *** You are using ptxas 10.0.145, which is older than 9.2.88. ptxas 9.x before 9.2.88 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
You do not need to update to CUDA 9.2.88; cherry-picking the ptxas binary is sufficient.
seems no problem for now
only thing left is python 3.7, which I haven't tried yet

*** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
I am facing this issue for Tensorflow==2.7.0 in ubuntu20.04. Please let me know how did you solve your issue.


Add required LABEL to Dockerfile.konflux

Pull Request Test Coverage Report for Build<PHONE_NUMBER>7
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall first build on update-dockerfile.konflux at 42.785%

Totals

Change from base Build<PHONE_NUMBER>2:
42.8%

Covered Lines:
3745

Relevant Lines:
8753

💛  - Coveralls

@astefanutti COuld you please review the PR?


Remote plugins

Pull Request Test Coverage Report for Build 717

23 of 43 (53.49%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.6%) to 72.087%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

weaveserver/services/core/appmgr.py
1
6
16.67%

weaveserver/services/plugins/plugins.py
22
37
59.46%

Totals

Change from base Build 705:
-0.6%

Covered Lines:
1126

Relevant Lines:
1562

💛  - Coveralls


bug--#493,will cause more to display the wrong number of items.

Expected Behavior

will cause more to display the wrong number of items.
#493
Current Behavior

Possible Solution

Steps to Reproduce

Screenshots

Browsers Affected

[ ] Chrome
[x] Firefox
[x] Edge
[ ] Safari 11
[ ] Safari 10
[ ] IE 11

Detailed Description

@stueja @hossainemruz

@Chow-Ray , can you please be a little more specific in your bug report as what you expected and what happens? Thanks.

I can confirm the issue. The solution is to start this counter from 1.
Ref: https://github.com/hugo-toha/toha/blob/99a131fe438870d16891623bda3634befa372d70/layouts/partials/navigators/navbar.html?_pjax=%23js-repo-pjax-container%2C div[itemtype%3D"http%3A%2F%2Fschema.org%2FSoftwareSourceCode"] main%2C [data-pjax-container]#L90

Should be fixed by https://github.com/hugo-toha/toha/pull/509


[berkeley] Remove ClientSdk and CompareSignatures jobs

!ci-build-me

!approved-for-mainnet

0-line PR, merging without review


cigien: Watch myseodr.com

cigien requests the watch of the watch_keyword myseodr\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

myseodr\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

myseodr\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Spevacus in Charcoal HQ

Approved by Spevacus in Charcoal HQ


Input not getting captured on hyprland with wayland feature

Bevy version
Bevy 0.13.2
[Optional] Relevant system information
Hyprland version 0.32.3
What you did
Using the wayland feature. Created this basic started app with default plugins, a system that runs every update to print "Hello!" and included a system to close the window on escape.
use bevy::prelude::*;

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Update, (hello, mouse_test, key_test))
        .run();
}

fn hello() {
    info!("Hello!");
}

fn mouse_test(mouse: Res<ButtonInput<MouseButton>>) {
    if mouse.just_pressed(MouseButton::Left) {
        info!("Mouse pressed!");
    }
}

fn key_test(keyboard: Res<ButtonInput<KeyCode>>) {
    if keyboard.just_pressed(KeyCode::KeyA) {
        info!("A pressed!");
     Additionally, the hello() system only triggers a few times before stopping completely. The number of times it triggers changes on each run.
Additional information
This issue does not occur if the wayland feature is not included.

Additionally, the hello() system only triggers a few times before stopping completely.

This makes me suspect that this is #10546. Can you run an example that moves on its own like animated_transform to confirm this?
cargo run --features wayland --example animated_transform

I get the same result as the user from the linked issue.

Okay, then we can close this as a duplicate.

Closing as a duplicate of #10546.


There are no commands defined in the "starter-kit" namespace.

Hi,
when i install the kit I get:
`  There are no commands defined in the "starter-kit" namespace.
In NewCommand.php line 348:
There was a problem installing Statamic with the chosen starter kit! `
it also happens with other starter kits.
what can I do?

You need to be running Statamic 3.2 :)


jvm: simplify prompt builder by using benchmark

This is similar to the C specific prompt builder. It's likely more fields will be used from the benchmark and this refactors things before we unnecessarily explode the number of arguments to the prompt builder.

/gcbrun exp -n dk-test-jvm126 -m vertex_ai_gemini-1-5 -b jvm

experiment is looking good


Offer a way to extend this documentation to avoid others from reproducing all of this good content :-)

This is a far fetched request :-). I work for a large company and we are migrating apps to Azure. We have generated a ton of documentation in wikis that cover this topic (private endpoints). At issue is that we have multiple people creating their own wikis, sharepoint sites, etc.  that are devoted at this topic. At our organization, we have a specific set of requirements that require engineers to uses various systems outside the Azure portal to setup private endpoints.
It would really be useful if we could have a way to simply augment your documentation with any specific requirements we may have. Obviously the end result would need to allow our organization to see their specific content in tandem with your content but prevent the public from seeing our content.
Again, I know this sounds far fetched but it is an extension of what I would like to see from application development in general. A technology that is emerging that seems to take a step in the direction is microfrontends through WebPack 5. That may not be the answer here but I am trying to fight a battle of folks continuously reinventing the wheel which turns into a nightmare to maintain.
An ideal solution would be that our company creates content that is specific to our needs for the private endpoint creation. That content is hosted in an app service and is made available as a remote plugin using Webpack 5. You can then present that extended content for those in our organization when they arrive on the page. Obviously an authentication scheme would need to be devised and more but that is my wish. An alternate approach would be  for your content to be plugged (served up as a remote plugible source) into our specific content.
Apologize for the long ramble but just looking for ways to eliminate all of the wikis we have generated and ultimately eliminate the tendency we all have as software engineers to keep reinventing the wheel.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b006a616-2f2f-8ecf-8b80-fd0a3cc62090
Version Independent ID: 92c54dc6-8ddf-b1f7-4a58-d61b8cd7fe25
Content: What is an Azure Private Endpoint?
Content Source: articles/private-link/private-endpoint-overview.md
Service: private-link
GitHub Login: @asudbring
Microsoft Alias: allensu

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. The requested updates have not been made since the creation of this issue, and the timeline for resolution may vary based on resourcing, so we've created an internal work item to incorporate your suggestions. We are closing this issue for now, but feel free to comment here as necessary. #please-close


gaali updated

gaali updated

This branch has conflicts that must be resolved
to resolve conflicts before continuing.
Conflicting files
api/gali.ts


Crash when autocrafting signalum fluxduct

Issue description:
Crash when autocrafting signalum fluxduct.
What happens:
Game crashes
What you expected to happen:
Signalum fluxduct gets autocrafted
Steps to reproduce:

Set up a complicated system to pulverise ingots, destabilize redstone, craft electrum and signalum blend, fill fluxducts, etc etc
Craft some stuff, cancel crafting, try again - that kind of thing

Version (make sure you are on the latest version before reporting):

Minecraft: 1.12.2
Forge: <IP_ADDRESS>73
Refined Storage:

Does this issue occur on a server? [yes/no]
If a (crash)log is relevant for this issue, link it here:
https://pastebin.com/wfgfceR0

Duplicate #1629


feat: added originalConversationId to payment report

@Vladimir-Petkov FYI: this is the more generic solution we came up with. We also load tested it for performance.


expose service in the from_file parser

Expose service in the from_file parser functionality allowing content-only or metadata-only extraction strategies, useful when trying to implement more intrincated OCR strategies from tika.
Have not changed from_buffer since it invokes CallServer directly.

Coverage decreased (-0.5%) to 42.492% when pulling febd892e066305d0128328993b8e6cb1ed02c1c3 on jjelosua:master into 8b88be2e731ad7d14e91b8475b2c15e70b0cca2c on chrismattmann:master.

@jjelosua can you please provide a test case along with this so I can verify there are no regressions? Thanks.

Will do


Resume

👷 Deploy Preview for rattandeep processing.

Name
Link

🔨 Latest commit
6c202a4a482f37760b1bb8da8c8a223c69e7eeb7

🔍 Latest deploy log
https://app.netlify.com/sites/rattandeep/deploys/659d760df9625d00084e7c4e


rclone mount on android tv box with no root

hi there,
i'm trying to mount a gdrive partition with rclone (1.47) on an android tv box (xiaomi mi box s).
i managed to access in ssh but i have no root privileges, and when i try to mount i get this:
2019/05/03 14:49:56 mount helper error: fusermount: failed to open /dev/fuse: Permission denied 2019/05/03 14:49:56 Fatal error: failed to mount FUSE fs: fusermount: exit status 1
i think it's because i don't have root privileges no change to make a su.
any idea how to solve this? it's sooooo annoying....
thanks a lot

This is a android limitation.  Not really anything rclone can address.  Even with root its hard to mount file systems and make them accessible across user spaces on android.

i think the issue is that rclone is trying to use fusermount that, by default, tries to use /dev/fuse where my user has insufficient rights.
is there any way to istruct rclone/fusermount to user another path where my user is the owner? sorry if it's a silly question, but i am not really a *nix user and i don't really know what fusermount or fuse are.
thanks

"This is a android limitation. Not really anything rclone can address. Even with root its hard to mount file systems and make them accessible across user spaces on android."
Nope.  You can't.

Thanks @calisro for providing the definitive answer.  I'll close this now - thanks.


Feature Request: Allow setting variables with YAML

Q
A

Issue Type
Feature Request

Deployer Version
4.3.0

Local Machine OS
macOS Sierra

Remote Machine OS
Ubuntu LTS 14.0.1

Description
It would be great to allow overriding or setting options with yaml like you can do for the server settings.
Here is an example config.yml:
application: myapp
default_stage: staging
deploy_path: /srv/users/serverpilot/apps/myapp
keep_releases: 5
local_uploads_dir: /Users/me/Sites/myapp/app/uploads
local_url: myapp.dev
public_dir: public
repository<EMAIL_ADDRESS>rsync_options: -avz --progress -e 'ssh -p {{server.port}}' --exclude=temp/
shared_dirs:
    - app/uploads
shared_files:
    - public/.htaccess
    - public/robots.txt
ssh_multiplexing: true
ssh_type: native
uploads_dir: shared/app/uploads
writable_dirs:
    - app/uploads

The function could look something like this:
namespace Deployer;

use Symfony\Component\Yaml\Yaml;

/**
 * @return void
 * @throws \RuntimeException
 */
function configList( $file ) {
    $configFileContent = Yaml::parse( file_get_contents( $file ));

    if( !is_array( $configFileContent )) {
        throw new \RuntimeException( "Error in parsing " . $file . " file." );
    }
    foreach( $configFileContent as $key => $value ) {
        set( $key, $value );
    }
}

And you could load both the deployer servers and config like this:
configList( 'config.yml' );
serverList( 'server.yml' );

I was thinking about creating new format for deployer: deploy.yml


Downloading OSO products?

Hello Rémi,
Does the tool allow downloading OSO products?
Thanks
Josselin

Hi Josselin,
I really don't know. If OSO is in the same catalog, then it should be doable theoretically. However for now only S2 level 2 & 3 models are implemented.

Ok merci !! Je vais trouver une autre solution !

Ok merci !! Je vais trouver une autre solution !


Assing role to multiple people

Hi,
I have a Bookstack instance and by default I am assigning "Viewer" role to all team members on sign-in using SSO. However I have created a new role "New_Role" and I want to assing this role to multiple people. I have atleast 200 users, out of which I want to assign this role to some 60-70 members. One way I see to do this is to login as admin and go to each user profile and manually add this role to them, but this is a hectic way.
Is there a way I can do it automatically by running some query or script. I have email Id's of all the users to whom I want to add this custom role.
Please help.

Depending on your SQL knowledge this could be done directly on the database, but Bookstack doesn't offer a native solution imho.
As far as I remember you have to fill the role_user table and run the php artisan bookstack:regenerate-permissions command afterwards. You should definitely make sure you have working backups before fiddling with the database directly.

Depending on what you are using for SSO and you already have the users you would like in the group you can sync the groups to each other.
In my instance i use Azure and you can setup the group sync by doing the following:
Create the group and write down the Object ID

Go to Roles in your bookstack instance and insert the Obejct ID into External Authentication ID

You will also need to add the following to your .env file in the bookstack directory

The groups link is not there by default so you will need to add it to your application

Thank you @cdrfun and @Mynster9361 for providing detailed answers to this. Will therefore close this off.


⚠ Bofrost.at has degraded performance

In 6485ca2, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1541 ms

Resolved: Bofrost.at performance has improved in 732a19e after 18 minutes.


Remove Bond 0.000 KPR (reflects not the right amount) can't remove bond

There is no way to remove your bond, it states 0.000 KPR instead of the correct bonded amount. So you can't unbond your bond. Please fix asap, I don't want to be slashed.

This is fixed by #12. Go check if your issue is fixed and then close this issue if so. Thanks!


Set custom timeout for connection to discord

Looks fine, please update the docs accordingly (including readme.md)


Support for offline mode user login

I wanted to know if offline user login is supported?

@raix Any update on this feature? Since this issue is a bit old, I wanted to check with you if an offline user-login method (authentication) is possible.

@niranjans persisting the user logged in when offline should be possible - encrypting via a pin code could be solved by building a layer on top of localforage and use that as storage.

I guess maybe this package can useful for this purpose: https://github.com/mbanting/meteor-cordova-accounts-resume/


Keyboard Manager issue on combinations

I changed my [,] and \ keys to number pad 2 5 8 accidentally. I used keyboard manager to change them back, but when I use shift + [, it becomes 2, and shift + ] becomes 5, shift + \ becomes 8. Tried shortcut remap, didn't work. Someone please help, I need that bracket to write programs for homework.

@Unforgiven6  click the "remap a key" button and delete the remappings and then click save

If you include the zip via the "Report Bug" option, may be able to see a bit more.
if you used a different app like (SharpKeys for instance), you'll have to reset them via that.


feat: login error info message

affects: @haulmont/jmix-front-generator, @haulmont/jmix-rest
Show addition information about Login errors

please rebase on master

please rebase on master

I did it. But CI failed with a strange error


Added the org chart web part

Q
A

Bug fix?
no

New feature?
no

New sample?
yes

What's in this Pull Request?
New SPFx sample the Org Chart web part using Office UI Fabric, React and OData batching

Hi @vman, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by Microsoft and real humans are currently evaluating your PR.
TTYL, MSBOT;