Refactor gitea internal event queues to use external library

Maybe we could use watermill library to replace (at least parts of) our queue code https://github.com/ThreeDotsLabs/watermill

There are something we have to consider:

looks like it doesn't support unique message
the message sequence is not guaranteed
there is no redis support which may break old version

Unique messages can be implemented as router middleware
Message sequence is really dependent on queue provider and mostly is not that important imho
This could be implemented and contributed as additional provider to library

I think we need specific reasons as to what is hoped to be gained and what's missing from queues.
It's one thing to suggest a drop-in work queue replacement. It's quite another to suggest completely refactoring.
From glancing at the repo this library is mostly focused on pubsub message queues. I think there's definitely something in being able to use pubsub for querying our work queues - but that is something different from what our work queues do.
For a start it really is worth noting that pub sub absolutely blocks a Redis connection - so you need one Redis connection per queue. At present we avoid this. This would limit the number of queues we could use.
It may well be that we could be inspired by watermill or even integrate it into our queue infrastructure but it doesn't seem suitable at first glance as a replacement. Instead this would require a either a complete refactor to use pure MQs or the creation of a separate MQ.
Now how could we use a pure MQ as a unique queue? Well we would have to either use a Redis/LevelDB Set on work retrieval or at message creation - or use some kind of DB locking on the records. That's a major refactor - potentially one that's worth it but it would likely mean that the simple API we currently have for the queues would go away. (Essentially our unique queue API as it stands is Push to the user and to the handler writer simply provide a handler.)
The alternative of adding a side channel event MQ with pubsub would allow us to keep our current queue infrastructure but allow the worker queue readers to be informed by messages on the event queue. I've been toying with the idea of a creating a broadcast queue for cluster events etc and a general MQ which could use pubsub to inform work queues but it seems likely that such a queue would be permanently alerted so wouldn't necessarily be particularly performant.
However I think you need to tell us why you think you'd want to do this and what you hope to gain.
We also need to have confidence that this project will still be there in 4, 2 - even 1 - year's time. Too many of Gitea's dependencies are in a poorly maintained state and we should be  thinking much more carefully before adding any more.

Hey @lafriks @zeripath, author of Watermill here!
If you are happy to explore deeper if Watermill may fit your use case I'm happy to assist. I'm also happy to look deeper to this RedisStream implementation (as it's not an official one). If the implementation quality would be good enough we could ask the author if it could be included in the list of official implementations.

From glancing at the repo this library is mostly focused on pubsub message queues. I think there's definitely something in being able to use pubsub for querying our work queues - but that is something different from what our work queues do.

Do you have any high-level overview of how you are using Redis? In general core library is pretty flexible on Pub/Sub model and it depends a lot on the specific implementation of Pub/Sub.

@roblaszczak thanks for getting in touch.
We're not currently using pubsub as the queues aren't really message queues - although they could be used that way and the intention of the queue infrastructure was to move us over time to a place where we could use event queues.
We have essentially two types of work queue - non-unique queues which use a Redis keyed List using RPush, LPop and LLen, and unique queues which wrap a Redis Set around the List. (using SAdd, SRem and SIsMember). The Redis lists are simply polled instead of using pubsub as that blocks redis connections and we're avoiding opening too many connections at present. (This is not ideal but appears to be functional.) We have leveldb which we use in a similar way.

I would propose that instead of using windmill directly immediately that we would simply create a windmill queue provider (at least in the first instance.) That would significantly minimize the size of PR to add this functionality and would make things a lot simpler. We'd need to think carefully about configuration and how it would be displayed to the user.
Allowing the non-unique work queues to use a windmill provider would be simple - create an event each time there is a push to a queue and wrap the message with an envelope of which work queue it was sent to. We could easily create a worker for each message type which could be one of our currently created dynamic worker pools or, if windmill supports its own we could let windmill do that itself.
Now, the unique work queues would be a bit more difficult - but I guess we could have a unique work queue handle their sets somewhere else and either have the unique message handler deal with the set or send a message back to a specific thing listening to remove things from the set.
(Presumably windmill has some lifecycle controls or allows subscribers to register and deregister on their own accord to allow us to correctly and safely manage shutdown of Gitea by shutting down workers independent of the brokers.)
One final consideration is that we may need some way of querying if the work queue is empty and to flush the work queue. The things that get pushed on to the work queue may unfortunately change between versions so it would be useful to flush them before shutting down for upgrade etc. I'm not sure if windmill has that functionality at a glance. If not we'd need to manage counts of items in each work queue separate from the messages similar to the sets above.

I think it would be quite simple to create a suitable redisstream provider and subscriber ourselves if the mentioned project wasn't ready - we'd likely need to do some of the work ourselves anyway. So the lack of a redis subscriber is not a problem. We'd also need to adjust our modules/nosql/manager_redis.go to give us a new client for each subscriber because of the above blocking.

The queue has been completely rewritten.
The only blocker is the Gitea's specialized "unique queue".
So I think this issue could be closed, the new code is much easier to maintain.


Only .md files processed

Makes is to that only files that end with ".md" are processed by the script.

Sure, that's reasonable. Thanks!


Synchronicity issues

Since the nature of our framework is asynchronous we need to provide ways to handle that situation.
Example
The user changes the cartocss of a layer and he wants to draw a red dot when there was an error. With the current api we got:
layer.setStyle(newStyle);
// layer fires  a ' styleChanged'  event without knowing the server's response

So the only way of detect an error would be through the client error api:
Notice this is not yet implemented and currently a cartoError has only a message field
client.on(carto.events.ERROR, function(cartoError) {
  switch(cartoError.type) {
    case: 'layerError':
     // ...
    break;
    ...
});

// If some error happened will be detected by the client error callback
layer.setStyle(newStyle);

Options

Callbacks
Promises
Events

Since callbacks are being deprecated and we must to change the API I bet for using promises and events resulting in a very powerful api.
layer.setStyle(newStyle)
  .then(nextFunction) // This callback will be executed once the style is set
  .catch(errorFunciton) // This callback will be executed if there was some errors

layer.on('error', callback); // The layer will also trigger errors
layer.on('success', callback); // Should we have a `success` event too?

This also alows using the async/await feature introduced in ES6.
try {
  await  layer.setStyle(newStyle);
  await  layer.setSource(newSource); // This method waits the style to be changed!
catch(cartoError) {
  errorFun(cartoError);
}

This is very useful to prevent unnecessary map reloads, think about an application where the user sets multiple changes into the map.
You could enqueue those changes and reload them at a stroke.
// This changes are not sent to the server
layer.setStyle(newStyle, {reload: false});
layer.setSource(newSource, {reload: false});
layer.setFeatureOverColumns(['foo'], {reload: false});
// Reload the map, once is reloaded show a popup, show widgets...etc
layer.reload()
  .then(() => popup('Reload ok!');
  .catch(cartoError => popup(cartoError.message); // Notice each layer/dataview will also throw its cartoErrors

Current status
We got a POC where the layer.setStyle and layer.setSource methods where updated
There is also a working example for updating layer styles.
TODO

[ ] Make dataview methods async
[ ] Implement layer success method?

I like the approach and agree that promises are very convenient cause you can check the result of the operation straight away. It can surely help.
My only concern here is having to document both promises and events. I just wonder if it will be a bit harder to explain/understand both things instead of having just events. Also, BUILDER works fine by only using events and we need events anyway, that's what makes me question this approach. Playing devils advocate a bit here.
@ivanmalagon @Jesus89 what do you guys think?

I think Promises are a nice way for the programmer to manage async operations. For example detecting success/error on a particular function or concatenate async methods.
Events are fine too, in order to detect changes without taking to account where the change happens, like in the dataviews' dataChange event for example.
So I think both approaches have advantages and they have complementary functionality so I would include both.

Let's give it a shot then!

@alonsogarciapablo @Jesus89 @ivanmalagon
So we agreed on a promise + events oriented API, we can get this with a few changes BUT:
IMO We need to address this first because with our design  we got some edge-cases (I explained some in slack) .
We could reduce those cases limiting our API.

Since we're combining promises and events, it would be nice if this error didn't appear:

So add a .reject callback in case something goes bad :P

Using promises should be optional IMO. We cant force everyone to add a .reject to every async method because then we are forcing them to use promises...

Using promises should be optional IMO

Is optional, but the browsers warns about an possible  uncaught error, not sure if we can prevent it...


RFP - Oculus Rift (CV1) software

Checklist

[x] The package I am requesting does not already exist on https://chocolatey.org/packages;
[x] There is no open issue for this package;
[x] The issue title starts 'RFP - ';
[x] The download URL is public and not locked behind a paywall / login;

Package Details
Software project URL : https://www.oculus.com/rift/setup/
Direct download URL for the software / installer : https://www.oculus.com/download_app/?id=1582076955407037
Software summary / short description: Drivers and software for Oculus Rift (original CV1 model)
The installer is a downloader for the actual install files so it might take some doing to automate.

Is it on the way ?

@pauby Any updates?

@TheCakeIsNaOH I'm going to remove myself from this. While I have an interest in it, it's going to sit here until I get time to do it. If you have the time to pick it up then it would seem to be better for you to do so.

Note: 21.7 GB are required to install Oculus.

I want to take over the work that @pauby did.
To do that I need to understand what has been done and what is the scope of work required.
please respond here on this and I would like to give it a go

@georgettica I have done no work on this at all. That's why I moved it back to being available for other maintainers.

@pauby ☺ all is good.
I'll look at similar projects and try to assess the depth of work before I dive into it.

seems like I have not time to work on it (as can be seen from the last comment I made) if someone aside of me wants to work on it the community will be happy

I'll pick this up again (I need it after rebuilding my VR machine).

I've looked into this and I'm going to pass on it. This isn't a good installer to package.
The setup is a stub installer (there is no offline installer)and runs you through a process including logging in etc. It doesn't have a silent option natively, I don't think AutoHotkey is going to be easy to script (although I'd imagine doable) and things like installing on a separate drive would have to be taken care of. IMO more work than it's worth.
I'm going to put this back to available for maintainers as I think it's doable, just a LOT of work.


 I've raised a PR with garble as they have a problem interpreting the go:linkname when reflection is being used.
You can see it here.
https://github.com/burrowers/garble/pull/883
I'll do a new release when they've fixed it, or if they take a little while I'll use my fork.

thanks bro for your swift answer

This is now on main, and I've updated the docker container. If you're using the binary install for the server, please install garble@master, rather than latest :)


[Suggestion] Make it possible to set the Reactor Port to automatically output steam

The reactor can produce insane amounts of steam by itself, but in Mekanism there is no way of transferring all of that steam to a turbine or a storage tank. If the port could be set to auto-output the steam, you would be able to directly connect it to a turbine, storage tank, or entangloporter and be able to take advantage of all of the Steam/tick that the reactor is generating. As of now the maximum output you can get with just mekanism is 44,800 mB of Steam/tick (Assuming that you have 1 laser focus matrix, 2 ports for Deuterium and Tritium input, 1 port for water input, and 1 port for direct energy output). This is less than the minimum steam/tick that the reactor produces.

already done, #3305 - use the configurator on a reactor port to change it to output mode. IMHO, this could be a bit more obvious, but I don't have an idea for that.

Oh I tried using a configurator and it didn't do anything. Maybe it was in the wrong mode.


Configuration Select Field : Fix (Options - Label)

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@adcac27). Click here to learn what that means.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff            @@
##             master     #104   +/-   ##
=========================================
  Coverage          ?   92.71%
=========================================
  Files             ?      119
  Lines             ?     1716
  Branches          ?      201
=========================================
  Hits              ?     1591
  Misses            ?      108
  Partials          ?       17


Attach "Today" to a channel/systemvariable

Hello,
would be very helpful if I could connect the state of "today"/"heute" with a channel or systemvariable. Would make this add-on much more helpful.
Georg

Hello Georg,
could you specify your suggestion?
Or do you mean this?
on("feiertage.0.today", "systemvariable"); // on change sync today with systemvariable

yes, thats what I mean - I know, I can do it with a script - but it would be easier to have it within the setup for "feiertage" especially for people who do not wanz to deal with scripts

Can you explain for which purpose someone would use a Systemvariable instead of the original adapter's datapoint "today"?

I would like to use the systemvariable within the programs on the CCU. Right now this is already done by a script - but, I'm running out of variables (max 200) and this script is using a lot of them. So I would like the Feiertage calculation being done in iobroker and the  result used on the CCU.
:) of course I can add the script to iobroker. But I thought making things easy is a good idea for people who cannot or don't want to deal with scripts in iobroker :)

I think this feature would be to special. I do not see the benefit.


Autocomplete

#24 > #20
Story
As a developer developing the application, I want a form autocomplete component named Autocomplete so that it can be used to capture data.
Description
A form autocomplete component is a user interface (UI) element that assists users in entering data by providing real-time suggestions or predictions based on their input.
Details
I'd like for you to implement a form autocomplete field component, this field will capture the input as is without validation or any other processing.
In addition, this component should not accept any props instead it should derive the props from FieldContext #77.
Acceptance Criteria

Project builds with no warning

Definition of Done

#4

Sub Tasks

[ ] Unit tests written
[ ] Integration test passing
[ ] Storybook stories written
[ ] Code peer-reviewed
[ ] Tested on supported devices/browser
[ ] Any configuration changes documented

Linked issues

Relates to
Linking type that allows you to establish a relationship between two issues.

#24

Blocked By
Issues or tasks that are blocking the progress of the issue.

#77

@wycliffepeart I added a description


Add typings for atom-keymap library

atom-keymap/atom-keymap.d.ts
check list

[ ] is collect naming convention?

https://www.npmjs.com/package/atom-keymap
http://bower.io/search/?q=atom-keymap
others?

[X] has a test file? (atom-keymap/atom-keymap-tests.ts or others)
[ ] pass the Travic-CI test?

Good catch, I've amended the previous commit to fix the project name & URL and force pushed.

@enlight :+1:


[SP-6506]-Backport of PDI-19752 - FTP Delete: the files are not deleted if it contains Japanese as a wildcard or full filename (9.3 Suite)

SP-6506-Backport of PDI-19752 - FTP Delete: the files are not deleted if it contains Japanese as a wildcard or full filename (9.3 Suite)

 Note:

Frogbot also supports Contextual Analysis, Secret Detection, IaC and SAST Vulnerabilities Scanning. This features are included as part of the JFrog Advanced Security package, which isn't enabled on your system.

🐸 JFrog Frogbot

:white_check_mark: Build finished in 25m 20s
Build command:
mvn clean verify -B -e -Daudit -Djs.no.sandbox -pl engine

:ok_hand: All tests passed!
Tests run: 2674, Failures: 0, Skipped: 6    Test Results

:information_source:  This is an automatic message


make rocm work with "ASUS TUF505DY"

I am the happy owner of the new "ASUS TUF505DY" notebook. it have the new "AMD Ryzen 5 3550H" (With Vega/Raven GPU) CPU and a "Radeon RX560X" GPU
I manage to instal Fedora29 (no success with Centos7...) I know it is not et suported OS... but i like to help
I will update all firmware ( picasso firmware) for Ryzen APU from the linux-firmware repo.
I build the last kernel (5.0.3) patching kfd_device.c with the id of the Raven:
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 8be9677c0c07..73e722cc6ae2 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -319,6 +319,7 @@ static const struct kfd_deviceid supported_devices[] = {
        { 0x9875, &carrizo_device_info },       /* Carrizo */
        { 0x9876, &carrizo_device_info },       /* Carrizo */
        { 0x9877, &carrizo_device_info },       /* Carrizo */
+       { 0x15D8, &raven_device_info },         /* Raven */
        { 0x15DD, &raven_device_info },         /* Raven */
 #endif
        { 0x67A0, &hawaii_device_info },        /* Hawaii */

after that the dmesg look good: (I am not sure it have to be done like this...)
> dmesg | grep kfd
[    2.492929] kfd kfd: Allocated 3969056 bytes on gart
[    2.494745] kfd kfd: added device 1002:67ef
[    2.653922] kfd kfd: Allocated 3969056 bytes on gart
[    2.654235] kfd kfd: added device 1002:15d8

after some install of centos rpm (rocm2.2) rocminfo report:
hsa api call failure at line 952, file: /home/philou/tmp/rocm/rocminfo/rocminfo.cc. Call returned 4104

after read some issus report I uninstall all rocm rpm and clone rocm repo (master branche):
> ROCT-Thunk-Interface
> ROCR-Runtime
> rocminfo

all are build with DEBUG for use with gdb.
I Path ROCT-Thunk-Interface to add device 1002:15d8 (Raven...)
#> ROCT-Thunk-Interface
diff --git a/src/topology.c b/src/topology.c
index c7f8a28..c4b9e98 100644
--- a/src/topology.c
+++ b/src/topology.c
@@ -200,6 +200,7 @@ static struct hsa_gfxip_table {
        { 0x69A3, 9, 0, 4, 1, "Vega12", CHIP_VEGA12 },
        { 0x69Af, 9, 0, 4, 1, "Vega12", CHIP_VEGA12 },
        /* Raven */
+       { 0x15D8, 9, 0, 2, 0, "Raven", CHIP_RAVEN },
        { 0x15DD, 9, 0, 2, 0, "Raven", CHIP_RAVEN },
        /* Vega20 */
        { 0x66A0, 9, 0, 6, 1, "Vega20", CHIP_VEGA20 },

and track rocminfo with gdb.
I finish in 'ROCT-Thunk-Interface'/src/openclose.c
HSAKMT_STATUS HSAKMTAPI hsaKmtOpenKFD(void)
{
	HSAKMT_STATUS result;
	int fd;
	HsaSystemProperties sys_props;

	pthread_mutex_lock(&hsakmt_mutex);

	/* If the process has forked, the child process must re-initialize
	 * it's connection to KFD. Any references tracked by kfd_open_count
	 * belong to the parent
	 */
	if (is_forked_child())
		clear_after_fork();

	if (kfd_open_count == 0) {
		init_vars_from_env();

		fd = open(kfd_device_name, O_RDWR | O_CLOEXEC);

// > fd = -1...
// > errno = EAGAIN

I find in 'drivers/gpu/drm/amd/amdkfd/kfd_device.c' it can be probleme with reset...
I don't know how to debug inside kfd kernel module (i need more time with google ;))
With dmesg I have many reset...
[   14.164505] amdgpu 0000:01:00.0: GPU pci config reset
[   29.774100] amdgpu 0000:01:00.0: GPU pci config reset
[   36.345021] amdgpu 0000:01:00.0: GPU pci config reset

dmesg-5.0.3.txt

OK I had some trace in "drivers/gpu/drm/amd/amdkfd/kfd_device.c" to track the lock...
dmesg give:
[    2.640380] kfd kfd: Allocated 3969056 bytes on gart
[    2.642222] kfd kfd: kfd_resume 1002:67ef
[    2.642590] kfd kfd: added device 1002:67ef
[    2.794867] kfd kfd: Allocated 3969056 bytes on gart
[    2.794964] kfd kfd: kfd_resume 1002:15d8
[    2.795207] kfd kfd: added device 1002:15d8
[   14.689643] kfd kfd: kgd2kfd_suspend 1002:67ef
[   14.689650] kfd kfd: kfd_locked++ 1002:67ef
[   28.151503] kfd kfd: kgd2kfd_resume 1002:67ef
[   28.151506] kfd kfd: kfd_resume 1002:67ef
[   28.151548] kfd kfd: kfd_locked-- 1002:67ef
[   37.695902] kfd kfd: kgd2kfd_suspend 1002:67ef
[   37.695908] kfd kfd: kfd_locked++ 1002:67ef
[   38.407090] kfd kfd: kgd2kfd_resume 1002:67ef
[   38.407092] kfd kfd: kfd_resume 1002:67ef
[   38.407133] kfd kfd: kfd_locked-- 1002:67ef
[   45.264709] kfd kfd: kgd2kfd_suspend 1002:67ef
[   45.264716] kfd kfd: kfd_locked++ 1002:67ef
[  164.478025] kfd kfd: kfd_is_locked => 1

if I don't make mistake:

1002:15d8  is the RAVEN eGPU
1002:67ef is the RX560X ...
so after some suspend/resume on the RX560... it finish with a suspend...
the last trace is when I call rocminfo => result in global lock => EAGAIN in open(/dev/kfd)

now I know what happen... but not why
That's all for today...

Hi @Djip007
I'll note that, at this time, iGPU + dGPU configurations are not supported in ROCm. As such, I suspect that that may be part of the underlying issue you're seeing. If you would like to use ROCm on your laptop, you may try disabling either the integrated or discrete GPU in your BIOS.

Ok some more news.
I know wy /dev/kdf is lock...
[philou@localhost rocm]$ cat /sys/bus/pci/drivers/amdgpu/*/power/control
dGPU > auto
iGPU > on
[philou@localhost rocm]$ cat /sys/bus/pci/drivers/amdgpu/*/power/runtime_status
dGPU > suspended
iGPU > active

the dGPU is suspend after 5s...
after activated :
echo on | sudo tee /sys/bus/pci/drivers/amdgpu/0000\:01\:00.0/power/control

it is much good... (but us 2W more power ;) ):
[zzzz@localhost rocm]$ ./rocminfo/build/rocminfo
LoadLib(libhsa-ext-finalize64.so.1) failed: libhsa-ext-finalize64.so.1: cannot open shared object file: No such file or directory
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:<PHONE_NUMBER>3709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32(0x20) KB
  Chip ID:                 5592(0x15d8)
  Cacheline Size:          64(0x40)
  Max Clock Frequency (MHz):2100
  BDFID:                   1280(0x500)
  Compute Unit:            8(0x8)
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    8388224(0x7ffe80) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Acessible by all:        TRUE
  ISA Info:
    N/A
*******
Agent 2
*******
  Name:                    gfx902
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          4096(0x1000)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
  Chip ID:                 5592(0x15d8)
  Cacheline Size:          64(0x40)
  Max Clock Frequency (MHz):1200
  BDFID:                   1280(0x500)
  Compute Unit:            11(0xb)
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      FALSE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Waves Per CU:            160(0xa0)
  Max Work-item Per CU:    10240(0x2800)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max number Of fbarriers Per Workgroup:32
  Pool Info:
    Pool 1
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Acessible by all:        FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx902+xnack
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*******
Agent 3
*******
  Name:                    gfx803
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          4096(0x1000)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
  Chip ID:                 26607(0x67ef)
  Cacheline Size:          64(0x40)
  Max Clock Frequency (MHz):1223
  BDFID:                   256(0x100)
  Compute Unit:            14(0xe)
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      FALSE
  Wavefront Size:          64(0x40)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Waves Per CU:            40(0x28)
  Max Work-item Per CU:    2560(0xa00)
  Grid Max Size:<PHONE_NUMBER>(0xffffffff)
  Grid Max Size per Dimension:
    x<PHONE_NUMBER>(0xffffffff)
    y<PHONE_NUMBER>(0xffffffff)
    z<PHONE_NUMBER>(0xffffffff)
  Max number Of fbarriers Per Workgroup:32
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    4194304(0x400000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Alignment:         4KB
      Acessible by all:        FALSE
    Pool 2
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Alignment:         0KB
      Acessible by all:        FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx803
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:<PHONE_NUMBER>(0xffffffff)
      Grid Max Size per Dimension:
        x<PHONE_NUMBER>(0xffffffff)
        y<PHONE_NUMBER>(0xffffffff)
        z<PHONE_NUMBER>(0xffffffff)
      FBarrier Max Size:       32
*** Done ***
rocminfo: /home/philou/tmp/rocm/ROCR-Runtime/src/core/runtime/amd_memory_region.cpp:68: static void amd::MemoryRegion::FreeKfdMemory(void*, size_t): Assertion `status == HSAKMT_STATUS_SUCCESS' failed.
Abandon (core dumped)

[zzzz@localhost rocm]$ clinfo
LoadLib(libhsa-ext-finalize64.so.1) failed: libhsa-ext-finalize64.so.1: cannot open shared object file: No such file or directory
malloc(): invalid size (unsorted)
Abandon (core dumped)

well like @jlgreathouse point it is an other probleme...

Next. I don't have option (as may other) have option to deactivate GPU in BIOS...
To "remove" dGPU I had a look on how to use pciePassTrough for VM... (https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU)
I build my kernel with "vfio_pci" buildin ...
#add in boot kernel param:
vfio-pci.ids=1002:67ef

and evry thing work (rocminfo & clinfo... test with dacktable)
I don't succes with activate only dGPU... but may be impossible lock like is use PRIM MUXless...

So next I'll try make rocm work with dGPU on centosVM with pcipassthrough...

Picasso id is on the way.
https://cgit.freedesktop.org/~agd5f/linux/commit/?h=drm-fixes-5.1&id=e7ad88553aa1d48e950ca9a4934d246c1bee4be4

After apply my patch (https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/issues/66#issuecomment-491582954) on kernel 5.1.1 on fedora 30 I manage to reinstall rocm (version 2.4)
what I did:

Add ROCm repos like for Centos.
insall:

dnf install rocm-opencl
dnf install rocm-smi
dnf install rocminfo
dnf install hsakmt-roct
dnf remove hsakmt

hsakmt need to be remove  => fedora have old version buildin (=> hsakmt.x86_64   1.0.6-8.rocm2.0.0.fc30  @fedora)
rocminfo is pik from fedora repo...
some come from ROCm repo other from fedora:
sudo dnf list installed | grep roc
hsa-rocr-dev.x86_64                               1.1.9_68_gc862c1cf-1                    @ROCm
hsakmt-roct.x86_64                                1.0.9_139_g126c93b-1                    @ROCm
rocm-opencl.x86_64                                1.2.0-2019050359                        @ROCm
rocm-smi.x86_64                                   1.0.0_129_g71924de-1                    @ROCm
rocminfo.x86_64                                   1.0.0-2.fc30                            @fedora

with that and default kernel param : "rocm_mode=3" (ie raven iGPU in use)
rocminfo & clinfo success

Note: 'dnf install rocm-dev' because of hcc depend
 Problème: package rocm-dev-2.4.25-1.x86_64 requires hcc, but none of the providers can be installed
  - conflicting requests
  - nothing provides pth needed by hcc-1.3.19174-1.x86_64

may be can be remove from this package dependency (can make sense on this deprecated package)

this will do to solve that problem:
https://copr.fedorainfracloud.org/coprs/fatka/pth/monitor/

Problème: package rocm-dev-2.4.25-1.x86_64 requires hcc, but none of the providers can be installed

conflicting requests
nothing provides pth needed by hcc-1.3.19174-1.x86_64

may be can be remove from this package dependency (can make sense on this deprecated package)

next test on fedora30:
thanks @sos-michael
install rocm lib for tensorflow
sudo dnf copr enable fatka/pth
sudo dnf install pth
sudo dnf install rocm-libs
sudo dnf install miopen-hip
sudo dnf install cxlactivitylogger

install rocm-tensorflow in virtual env.
and jupyter notebook (for simple test.)
python3.7 -m venv ./rocm
source ./rocm/bin/activate
pip install --upgrade pip
pip install tensorflow-rocm
pip install jupyter notebook

start jupyter?
jupyter notebook

test sample code:
import tensorflow as tf

hello = tf.Variable("hello world")

sess = tf.Session()
init = tf.global_variables_initializer()

sess.run(init)
sess.run(hello)

result for dGPU
=>
WARNING:tensorflow:From /home/philou/Developpement/python/rocm/lib64/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.

b'hello world'

It work... (yes I know... et very simple test...)
but on iGPU it failed:
FailedPreconditionError: Failed to memcopy into scratch buffer for device 0

not sure if it have or not... but look like it try to alloc to much memory (31Go of my 32Go...) more than the 3/8 allowed... don't know if it can be correct... and where...
any way not that bad for this fedora test.
name: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
AMDGPU ISA: gfx902
memoryClockRate (GHz) 1.2
pciBusID 0000:05:00.0
Total memory: 32.00GiB
Free memory: 31.75GiB
2019-05-28 23:07:05.516984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1642] Adding visible gpu devices: 0
2019-05-28 23:07:05.516995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-28 23:07:05.517000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1059]      0
2019-05-28 23:07:05.517006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1072] 0:   N
2019-05-28 23:07:05.517047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1189] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30886 MB memory) -> physical GPU (device: 0, name: AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx, pci bus id: 0000:05:00.0)
2019-05-28 23:07:09.110206: E tensorflow/stream_executor/rocm/rocm_driver.cc:461] failed to memset memory: HIP_ERROR_InvalidValue

@Djip007 Did the PCI passthrough disable the dGPU or the iGPU ?
I feel that it would be more beneficial to have a switch similar to DRI_PRIME to change the active GPU depending on the need. Is this feasible ?

@benkenobi007 sorry did not see your post..
pci path through disable GPU for the host only VM guest can use it.
For DRI_PRIME like I have a patch here: (but you have to chose at boot-time)
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/issues/66#issuecomment-491582954
It make chose if iGPU or dGPU is assigne to rocm at boot time. that way both GPU is usable for OpenGL/Vulkan...

@Djip007 I have the same laptop. And I tried to install rocm on Ubuntu 19.04. Resault is broken system.
Problem is absent support from AMD for rx560x on linux systems.
I wrote to amd on this topic, they answered like this...

Thank you for the email,
Currently we donot have the support, At this point of time I cannot comment on the future releasing,I will make a note of this issue and report to the driver team
Thanks for contacting AMD

Do you have any good news?

RX560X work with rocm, problem is with the mix on iGPU+dGPU.
I dont know which kernel use Ubuntu (I use Fedora) ...
More info/patch here:
https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/issues/66#issuecomment-520846346
May be you can point it to AMD...
Anyway can you report some of your error log to be sur it is the same problem?

Unfortunately not. I delete my Ubuntu distro.
But, are you sure on Fedora Rocm is work properly?  If it is maybe possible to launch Davinci Resolve on Fedora now, because without it, I've got crash on start...

use it like with tensorflow like that:
podman run -it
--network=host
--device=/dev/kfd
--device=/dev/dri
--ipc=host
--shm-size 16G
-v $PWD:/root/home
rocm/tensorflow:latest
https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/issues/935
(with the patch: https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/issues/66#issuecomment-615980464 ...)

Hi @Djip007
Is the issue gone with the recent releases? Please share some update on this.
Thank you.

Hi @Djip007
Is the issue gone with the recent releases? Please share some update on this.
Thank you.

Sorry for the question @Djip007 , but ill gonna install Fedora because on Ubuntu 20.04 ROCM dont run in this laptot... I am very noob in Linux, can u explain me how i use the patch you upload on Fedora 33? and this patch automatically take the RX560X and disable Vega?. Need another driver to install maybe or just be okey with POLARIS 11? and need install the package on rocm, because in rocm documentation page say " newer versions of Ubuntu may not be compatible with the rock-dkms kernel driver. In this case, you can exclude the rocm-dkms and rock-dkms packages" and this version on UBUNTU use a Kernel 5.8
Thanks for the time to answer me , Have a great day

Sorry for the question @Djip007 , but ill gonna install Fedora because on Ubuntu 20.04 ROCM dont run in this laptot... I am very noob in Linux, can u explain me how i use the patch you upload on Fedora 33? and this patch automatically take the RX560X and disable Vega?. Need another driver to install maybe or just be okey with POLARIS 11? and need install the package on rocm, because in rocm documentation page say " newer versions of Ubuntu may not be compatible with the rock-dkms kernel driver. In this case, you can exclude the rocm-dkms and rock-dkms packages" and this version on UBUNTU use a Kernel 5.8
Thanks for the time to answer me , Have a great day

Hi,
I have similar situation here with my laptop which have vega 8 iGPU and RX560X dGPU. So I am unable to use rocm for this machine. I am using Linux mint 20 and new to all the kernel building stuff. I have tried building kernel for myself and patching with @Djip007 's patches for 5.9.x. I could not boot with that kernel as it is rebooting whenever I try. I have also tried to install different versions of rocm with no luck of get it working. Any idea about make it work in linux mint ? I cannot change the distro for the time being as it would affect my work flow.

Has this issue been resolved?  If so, can we please close it?

Original ticket is more than a year old and the person that opened ticket originally has not responded to the latest request. If this is still an issue, please file a new ticket and we will be more than happy to investigate it. Thanks!


[QUESTION] the denpendecy node task is still running when the task in the dependency has finished success

version 1.1.0
the denpendecy node task is still running when the task in the dependency has finished success.
A工作流里有个B工作流的依赖型节点，当B工作流已经成功运行完成时，A工作流里的依赖型节点还在执行，这是不符合预期的。

Can we reproduce this scene so that we can locate what the problem is

能复现这个场景么? 方便我们定位问题！

I know the problem. It is because there is one  dependency task is still running，but not the latest one，the latest one was success. Doesn't the dependency task node check the latset instance was success?
找到原因了。我的依赖节点是一个5分钟调度一次的工作流，有个工作流实例一直处在运行状态，但之后的调度都成功了。依赖节点不是应该检测的是最后一次运行实例有没有成功吧？


⚠ Changes to make IPAM deployable with CAPI

These changes allow metal3 ip-address-manager(IPAM) to be deployed with cluster-api (CAPI). Currently deploying IPAM with CAPI assumes the CRDs are in a git release and the release has a metadata.yaml.  IPAM's current release doesn't have a metadata.yaml so this need to be added. Action need to be taken to get IPAM as an ipam-provider for CAPI so that metal3IPAM can be specified when running clusterctl init with the --ipam flag.
IPAM's ipam-components.yaml has CRDs for ipClaim and ipAddresses which work but IPAM is unable to resolve the ipAddresClaims that CAPI has defined. This is a next step after these changes are
meged to make IPAM resolve CAPI's ipAddressClaims. Check more in CAPI ipam contract

/cc @lentzi90 @zaneb @kashifest @schrej @mboukhalfa

/cc @furkatgofurov7

@furkatgofurov7 @schrej can we get some reviews from you folks on this ?

@peppi-lotta can you update the description? The part about "in-cluster directory" was a misunderstanding if I recall correctly

Imho we should avoid deploying custom claims and addresses when using clusterctl to install metal3 ipam. If the plan is to support the CAPI variants of those resources anyway, deploying the custom resources is probably superfluous, and will only lead to confusion (and annoying autocompletion). As long as the official resources aren't supported, I'd also not accept the addition of the metal3 ipam to the official list of providers integrated in clusterctl, as it's not compatible with other providers. Not sure if the CAPI team has the same opinion here.

I want to understand this part a bit more. I can agree to the point that metal3 ipam needs to handle the CAPI variant of the IPAM resources to be accepted as a CAPI IPAM provider. But on the point of deploying metal3 specific resources and CRDs, it should remain as it is now since if a user want to deploy a metal3 IPAM then the user is aware that metal3 IPAM allows you to have this custom claims on top and offers you some additional workflow. So I dont see any harm deploying those. Otherwise if someone doesnt want metal3 IPAM but only CAPI specific variants then the current IPAM provider should be enough.

What is the current state of this ticket ? @peppi-lotta  could you please summarize it here? The PR went silent in February.

These changes along https://github.com/metal3-io/cluster-api-provider-metal3/pull/1442  help make it possible to deploy this ip-address-manager with clusterctl. This has been on hold while I've been researching the possibility to switch to use in-cluster-ipam in CAPM3. This is currently not possible so I'll continue to make this ip-address-manager an IPAM-provider for CAPI. I'll hopefully be able to continue with this soon.

/remove-lifecycle stale


Automatonymous scheduling

I am facing an issue where an event is being scheduled twice by quartz even though the state machine has only one event where that can schedule another.

Are you using Publish for your message scheduler, and perhaps have two separate instances of your quartz scheduler on two separate queues? If you're seeing double, look for doubles in your system that correlate, and see if it's the cause.

Resolved the issue already.But Thanks.However, i need further help. Need to have a custom schedule trigger for scheduling events for the schedule. For example, I have a schedule which needs to be triggered after a,b and c seconds but i am not able to do it via scheduling extensions.
One or more of these triggered events also need to be unscheduled based on certain condidtion

Detailed explanation: My State machine is about user subscription. Purchased and Cancelled are the events.When subscription is purchased, i need to schedule another event(say, for sending email to customers for feedback) 3 times in future : after 1 month, 3month and 7 month. A cancellation event should unschedule all forthcoming scheduled events except the 1 month one.

@phatboyg : Can you explain a bit on how i can achieve this or provide me some link ?

You can use more than one schedule in a state machine, sounds like you might need more than one schedule.

@phatboyg : But how do i code up the Received Event for the same. It only receives the last scheduled event (since schedule is same only delay is different)and throws exception. AnyReceived Event too throws exception:
public SubscriptionStateMachine()
{
InstanceState(x => x.CurrentState);
        Event(() => Purchased, x => x.CorrelateBy(sub => sub.SubscriptionId.ToString(), context => context.Message.SubscriptionId.ToString()).SelectId(context => Guid.NewGuid()));

        Event(() => Cancelled, x => x.CorrelateBy(sub => sub.SubscriptionId.ToString(), context => context.Message.SubscriptionId.ToString()));

        Schedule(() => Surveyed, x => x.ExpirationId, x =>
        {
            x.Delay = TimeSpan.FromSeconds(5);
            x.Received = e => e.CorrelateById(context => context.Message.CorrelationId);
        });

        Initially(
            When(Purchased)
                .Then(context => Console.Out.WriteLineAsync($"subscription purchased: {context.Data.EmailId} to {context.Instance.CorrelationId}"))
                .Then(context =>
                {
                    context.Instance.ActivationDate = context.Data.ActivationDate;
                    context.Instance.EmailId = context.Data.EmailId;
                    context.Instance.Channel = context.Data.Channel;
                    context.Instance.CountryCode = context.Data.CountryCode;
                    context.Instance.CurrencyCode= context.Data.CurrencyCode;
                    context.Instance.LanguageCode = context.Data.LanguageCode;
                    context.Instance.Name = context.Data.Name;
                    context.Instance.OrderId = context.Data.OrderId;
                    context.Instance.SubscriptionName = context.Data.SubscriptionName;
                    context.Instance.SubscriptionType = context.Data.SubscriptionType;
                    context.Instance.SubscriptionId = context.Data.SubscriptionId;
                    context.Instance.IsCancelled = false;
                })
                //.ThenAsync(context => Console.Out.WriteLineAsync($"subscription purchased: {context.Data.EmailId} to {context.Instance.CorrelationId}"))
                .Schedule(Surveyed, context => new SurveyScheduleEvent(context.Instance))

                .TransitionTo(Active)
            );

        During(Active,
            When(Cancelled)
                .Then(context => Console.Out.WriteLineAsync($"subscription cancelled: {context.Data.SubscriptionId} to {context.Instance.CorrelationId}"))
                .Then(context =>
                {
                    context.Instance.SubscriptionId = context.Data.SubscriptionId;
                    context.Instance.IsCancelled = true;
                    context.Instance.CancellationDate = context.Data.CancellationDate;
                })
                //.ThenAsync(context => Console.Out.WriteLineAsync($"subscription cancelled: {context.Data.SubscriptionId} to {context.Instance.CorrelationId}"))
                .Unschedule(Surveyed)
                .TransitionTo(InActive),

              When(Surveyed.Received)
                .Then(context => Console.Out.WriteLineAsync($"Surveyed through delighted now {context.Instance.CorrelationId}"))
                //.ThenAsync(context => Console.Out.WriteLineAsync($"Surveyed through delighted now {context.Instance.CorrelationId}"))
                .Publish(context=>new SubscriptionChangeEvent(context.Instance))
                .Finalize()
            );

       SetCompletedWhenFinalized();
    }

    public State Active { get; private set; }
    public State InActive { get; private set; }

    public Schedule<Subscription, SurveySchedule> Surveyed{ get; set; }

    public Event<SubscriptionPurchased> Purchased { get; private set; }
    public Event<SubscriptionCancelled> Cancelled { get; private set; }

If only the delay is different, you can pass a custom delay within the .Schedule() method itself:
.Schedule(context => TimeSpan.FromSeconds(30), new Message())

Sorry, you still need the schedule property as the first argument.
.Schedule(MySchedule, context =>..., new Msg());

@phatboyg : so what you are suggesting is .Schedule(MySchedule, context => TimeSpan.FromSeconds(30), new Message())  ?

Yep

This line is an example of what I'm talking about:
https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit.AutomatonymousIntegration.Tests/ScheduleTimeout_Specs.cs#L253
The delay is that last argument in the parameter list.

@phatboyg : The example you specified is very similar to what i intend to do and complies without error. However, the actual time delay when running the state machine is same as the timeDelay specified in Schedule initialization:
Schedule(() => CartTimeout, x => x.CartTimeoutTokenId, x =>
{
x.Delay = TimeSpan.FromSeconds(30);
x.Received = p => p.CorrelateBy(state => state.MemberNumber, context => context.Message.MemberNumber);
});
ie, in the example you menationed, the timeout is still 30 seconds and not 3 seconds as intended.

Did you figure this out, so it can be closed?


added auto placing option (issues #71)

broke down the setPosition() function into two-part. The calculation for the hostStyleTop and hostStyleLeft is taken into another function named setHostStyle() and auto placing is configured by trying 'right', 'bottom', 'top', 'left' one by one.
added a draggable button to test the feature.

@wmamunhossain Cool!


Disallow subfolders in the model package

We didn't specify how the model package should behave when it comes to subfolders. Since Zenodo does not allow folders, and also in general it's easier to just do a plain folder of files without subfolders in the model package.
I hereby propose that we write in the spec that a model package must not contain any subfolder. If necessary, they can use zip files in side the model pacakge, same as we did with the deepimagej model weights.

Update: although zenodo does not allow subfolders, but it allows / in the file name, this will allows us implementing virtual folders. We need to implement this in the packagers.
Basically we need to download the file and check whether its file name contains /. If yes, we place it under the corresponding folder structure.

[ ] implement this in python packager @FynnBe
[ ] implement this in js packager @oeway

So maybe we can allow subfolders, but definitely not allow to refer to parent folder.

/ in the file name sounds very error-prone.. could we not just keep it flat and substitute / with _?

/ in the file name sounds very error-prone.. could we not just keep it flat and substitute / with _?

I would also be in favor of keeping it flat.

just tested uploading a nested zip to zenodo...
did you already implement it in the js packager, @oeway ?

(it's the 'nested/folder/README.md')

btw, a click on the "Download RDF Package" link

gives me

That's by design, right? What needs to happen for it to work? could we add that in the 'error' message?

fyi: the uploaded test package is broken...

/ in the file name sounds very error-prone.. could we not just keep it flat and substitute / with _?

Technically, that's not a file name, it's a key or a path, it's common for s3-like storage systems and it allows supporting virtual folders.
This means it will have no impact on our spec at all, it only affect how we download and upload from zenodo.
It won't matter much for model packages if we only allow flat folder structure, but it will make a difference for dataset pacakage for example, we can then support zarr directory store stored in a subfolder.

That's by design, right? What needs to happen for it to work? could we add that in the 'error' message?

Currently, that link will only work if we approve it to be part of bioimage.io, I assume your new model is not approved yet, right? We can perhaps also support the link even if it's approved yet, but we might want to show a warning to the user that this package is not approved yet.

When trying to update my test entry (https://sandbox.zenodo.org/record/892181#.YQkSlo5MFjG)
I cannot edit the 'documentation' field.... Is this a bug or just a missing feature?

ah, I suppose I could in 'Review & Upload'... yeah that's fine! 👍

alright, with some fixes it works:
>>> raw_model = spec.load_raw_resource_description("https://sandbox.zenodo.org/record/892199/files/rdf.yaml")
>>> raw_model.documentation
URI(scheme='https', authority='sandbox.zenodo.org', path='/record/892199/files/nested/folder/README.md', query='', fragment='')
>>> model = spec.load_resource_description("https://sandbox.zenodo.org/record/892199/files/rdf.yaml")
>>> model.documentation
WindowsPath('C:/Users/fbeut/AppData/Local/Temp/bioimageio_cache/https/sandbox.zenodo.org/record/892199/files/nested/folder/README.md')

Currently, that link will only work if we approve it to be part of bioimage.io, I assume your new model is not approved yet, right? We can perhaps also support the link even if it's approved yet, but we might want to show a warning to the user that this package is not approved yet.

Maybe we can just mention that in the 'oops: resource not found' pop-up...

fyi: the uploaded test package is broken...

Which one is that?

Which one is that?

This one:

When trying to update my test entry (https://sandbox.zenodo.org/record/892181#.YQkSlo5MFjG)

whereas this (uglily just reuploaded and not updated) one works: https://sandbox.zenodo.org/record/892199

One cannot even delete from sandbox.zenodo, huh?

One cannot even delete from sandbox.zenodo, huh?

No, you have to forget about it after publish on sandbox zenodo.

whereas this (uglily just reuploaded and not updated) one works: https://sandbox.zenodo.org/record/892199

Is this a bug of the bioimage.io?

Is this a bug of the bioimage.io?

not necessarily. the documentation field cannot be changed in the json schema interface of the upload page and only in the following 'Review & Upload' step. I just realized this right after republishing the test model...
I suppose it would be nice if the first upload page would already give you somehow the possibility to select a documentation file and not just allow you to upload any files and then add them manually to the yaml in the next step...


Attempt to clear the BufferedReader class from IndexError

Addresses the following issues:

#69
#25
#20
#10

Continues the PR #90.
The assumption is the case when len(self.buffer) == self.position. Currently only two cases were caught:

self.buffer=b"" and self.position = 0 - can be when realising the logic of data reading;
when self.buffer is not empty and self.position = len(self.buffer) -> no reproducible example to lay out.

During the investigation, some refactoring stuff was brought.

@long2ice , hello, could you give your opinion on the PR and provide answers to the comments on the test cases described in the docstrings?
Also, could you tell are you planning to release a new version with, at least, this #90 PR to be included?

@long2ice

@long2ice , could you have a look at this PR and tell whether it is OK?


Spaces must be encoded as "+"

The encode function encodes a space as %20, not + as required.
> encode $ fromArray
>   [ Tuple "hey" Nothing
>   , Tuple "Oh" (Just "Let's go!")
>   ]
Just "hey&Oh=Let's%20go!"

Expected result:
Just "hey&Oh=Let's+go!"

W3C
https://www.w3.org/TR/html401/interact/forms.html#h-<IP_ADDRESS>

application/x-www-form-urlencoded Forms submitted with this content type must be encoded as follows: <...> Space characters are replaced by `+', and then reserved characters are escaped as described in [RFC1738] <...>

FormURLEncoded.purs
https://github.com/purescript-contrib/purescript-form-urlencoded/blob/v6.0.1/src/Data/FormURLEncoded.purs
Instead of functions encodeURIComponent  and decodeURIComponent, should use functions encodeFormURLComponent and decodeFormURLComponent.
33: -- | Encode `FormURLEncoded` as `application/x-www-form-urlencoded`.
34: encode :: FormURLEncoded -> Maybe String
35: encode = map (String.joinWith "&") <<< traverse encodePart <<< toArray
36:   where
37:     encodePart = case _ of
38:       Tuple k Nothing -> encodeURIComponent k
                             ^^^^^^^^^^^^^^^^^^
39:       Tuple k (Just v) -> (\key val -> key <> "=" <> val) <$> encodeURIComponent k <*> encodeURIComponent v
                                                                  ^^^^^^^^^^^^^^^^^^       ^^^^^^^^^^^^^^^^^^
40:
41: -- | Decode `FormURLEncoded` from `application/x-www-form-urlencoded`.
42: decode :: String -> Maybe FormURLEncoded
43: decode = map FormURLEncoded <<< traverse decodePart <<< String.split (Pattern "&")
44:   where
45:     decodePart = String.split (Pattern "=") >>> case _ of
46:       [k, v] -> (\key val -> Tuple key $ Just val) <$> decodeURIComponent k <*> decodeURIComponent v
                                                           ^^^^^^^^^^^^^^^^^^       ^^^^^^^^^^^^^^^^^^
47:       [k]    -> Tuple <$> decodeURIComponent k <*> pure Nothing
                              ^^^^^^^^^^^^^^^^^^

Thanks! That looks like the right change. Are you interested in sending a PR?

Yes, I will send a PR today.


IndexOutOfRangeException at line 141 of DotnetPathResolver.cs

Context:
This code used to work prior to recent VS and framework updates (I don't remember the specifics of the updates which were recommended by VS 2022). It walks through .csprojs and builds them with the following options:
            var envOptions = new EnvironmentOptions
            {
                EnvironmentVariables =
                {
                    ["IntermediateOutputPath"] = Path.Combine(rootSlnDir, "ba", "obj", project.ProjectGuid.ToString()),
                    ["OutputPath"] = Path.Combine(rootSlnDir, "ba", "bin", project.ProjectGuid.ToString())
                }
            };
           return project.Build(envOptions);

The error appears to be due to incomplete information in dotnet --info when run from within ProcessRunner:
Host (useful for support):
  Version: 6.0.2
  Commit:  839cdfb0ec
.NET SDKs installed:
  No SDKs were found.
.NET runtimes installed:
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.22 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.2 [C:\Program Files (x86)\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.22 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.2 [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files (x86)\dotnet\shared\Microsoft.WindowsDesktop.App]
To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Running from within Powershell produces more complete output:
.NET SDK (reflecting any global.json):
 Version:   6.0.200
 Commit:    4c30de7899

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.200\

Host (useful for support):
  Version: 6.0.2
  Commit:  839cdfb0ec

.NET SDKs installed:
  3.1.100 [C:\Program Files\dotnet\sdk]
  3.1.109 [C:\Program Files\dotnet\sdk]
  3.1.416 [C:\Program Files\dotnet\sdk]
  6.0.101 [C:\Program Files\dotnet\sdk]
  6.0.200 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.22 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

Any help would be appreciated. Thanks!

This part is particularly interesting:

No SDKs were found.

I've never seen that before, and it doesn't even really make sense to me from a dotnet --info call - I'm not sure how anything to do with the environment would hide the SDKs.
Is it possible you've encountered this problem or something like it?: https://github.com/dotnet/installer/issues/12616

@daveaglick Yep, and the first comment about deleting the x86 folder solved it for me. In hindsight, I suppose the big clue was in the output I pasted above. Notice that the first list of runtimes points to the Program Files (x86) folder while the second points to the regular Program Files folder. So, it seems like the tool gets tripped up when dotnet.exe is installed for more than one architecture.

Cool - glad it's resolved for you. This looks like an upstream bug with dotnet itself, which makes sense given that Buildalyzer just shells out to that (and relies on what it outputs to figure things out). Since this is an exceptional situation, unless you think there's something we can do on this end, I'm going to close it as out of scope for Buildalyzer.


Quality Changer

You can add this feature?

this is simple a download script which mean no playback, impossible task

Ok,  thanks


Add ServiceCIDR to IPAM config + add route from host in order to acce…

#558

Coverage increased (+0.002%) to 76.781% when pulling 443e4373032963d3d529f142813ef9de962ab74b on lukasmacko:master into 9e6fc4dea5dbb7f0d384ac2f8202697e8c48865c on contiv:master.

Please brief describe what is #558 and why this fixes #558 in commit msg

This patch fixes issue #558 (https://github.com/contiv/vpp/issues/558).
The solution is to add a default CIDR subnet from which Cluster IPs are allocated and to install a route on every host pointing to this subnet.


Inventory#getItem not working when it is an inventory with a result slot

Is there an existing issue for this?

[X] I have searched the existing issues

Are you using the latest version of MockBukkit?

[X] I am using the latest version of MockBukkit.

Minecraft Version
1.20
Describe the bug
I am testing player click behaviour and noticed, that the clicked slot in a WorkbenchInventoryMock does not behave as expected. I am trying to verify the result of the crafting table by checking index 0 in the contents of the inventory. In the current implementation the item at the underlying index from the array is returned. Now the array is filled only by the matrix, the result slot is ignored and instead has its own field in the class.
Reproducible Test
    @Test
    public void testInventoryResultClick() {
        Inventory inventory = stickCrafting(1);
        player.openInventory(inventory);
        player.simulateInventoryClick(player.getOpenInventory(), ClickType.LEFT, 0);
        assertEquals(new ItemStack(Material.STICK, 4), ((CraftingInventory) inventory).getResult()); // passes
        assertEquals(new ItemStack(Material.STICK, 4), inventory.getItem(0)); // fails
    }

    private static Inventory stickCrafting(int amountPerPlank) {
        WorkbenchInventoryMock workbenchInventoryMock = new WorkbenchInventoryMock(null);
        workbenchInventoryMock.setMatrix(new ItemStack[]{
                null, null, null,
                null, plank(amountPerPlank), null,
                null, plank(amountPerPlank), null
        });
        workbenchInventoryMock.setResult(new ItemStack(Material.STICK, Math.min(amountPerPlank*4, 64)));
        return workbenchInventoryMock;
    }

    private static ItemStack plank(int amount) {
        return new ItemStack(Material.OAK_PLANKS, amount);
    }

Anything else?
I know that the test itself is easily fixable by just casting it to CraftingInventory and then accessing the result. However, I am using the CraftingInventory in parameterized tests and I would prefer all inventories behaving similar.
A potential fix could be by overriding Inventory#getItem in WorkbenchInventoryMock and checking if the index is 0. If so, then return the result variable. If not, then return the ItemStack from the underlying array at index-1.

Currently I don't know how to simulate a player clicking in the result slot.

I could try to implement this behaviour along with support for the SlotType enum to add more functionality to the simulateInventoryClick methods. Potentially in the structure of #972 ?

I could try to implement this behaviour along with support for the SlotType enum to add more functionality to the simulateInventoryClick methods. Potentially in the structure of https://github.com/MockBukkit/MockBukkit/pull/972 ?

I failed to notice this issue (we should have an alert on issus opening in our discord). Anyhow, yes it's probably better to follow the same structure of #972 , but I would think that you should make the pull request to the most recent release.


Allow vector-2 for concatenate op when mem-intensive-opt is on.

mark related to https://github.com/alibaba/BladeDISC/issues/766


Support multiple elements in checkout and source-checkout

See original issue on GitLab
In GitLab by [Gitlab user @cs-shadow] on Nov 19, 2018, 16:05
Background
As pointed out in https://gitlab.com/BuildStream/buildstream/merge_requests/820#note_111756025, it might make sense to support multiple elements in the newly added source-chechout command. Similarly, it would also make sense to support multiple elements in the checkout command as well.
Possible solutions
The current format for both these commands is bst COMMAND ELEMENT DIRECTORY. Depending on how we choose to do it, it may or may not be an API break. I am sure there are other options, but I can see the following two options at present:

bst COMMAND ELEMENT [ELEMENT..] --dir DIRECTORY
bst COMMAND ELEMENT DIRECTORY [-e ELEMENT DIRECTORY...]

The first option will be similar to how workspace open will soon handle multiple elements, see !897. But, for it to work properly, we may first need to add an option for configuring default location for checking out elements. Alternatively, we can support --directory for multiple elements by creating directories named after each element and checking it out under that subdirectory. (This is similar to how source-checkout currently handles dependencies.
EDIT: The consensus seems to be in favor of this option.
The second option option avoids breaking the API but is more clunky IMO. That said, it is similar to how multiple elements are planned to be handled in bst shell, see !909.
Task description

[ ] Support multiple elements in bst checkout/bst artifact checkout
[ ] Support multiple elements in bst source checkout

Acceptance Criteria
It is possible to checkout multiple elements (or sources in case of source-checkout) using a single bst invocation.

In GitLab by [Gitlab user @cs-shadow] on Nov 19, 2018, 16:09
marked this issue as related to #362

In GitLab by [Gitlab user @cs-shadow] on Nov 19, 2018, 17:10
changed the description

In GitLab by [Gitlab user @cs-shadow] on Dec 7, 2018, 15:05
See https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html and its follow-ups for related discussion.

In GitLab by [Gitlab user @cs-shadow] on Dec 18, 2018, 12:25
changed the description

I find myself in disagreement with this issue.
From my perspective:

The original thread speaks of the ability of bst workspace open to handle multiple targets, but this is rather different since workspaces always only apply to a single element
The checkout commands in question both have --except and --deps options, which means the proposed syntaxes in the issue would not be able to handle those specifics for each target being checked out
There does not appear to be much in the way of material gain in supporting multiple targets in the checkout routines (given that these are IO intensive tasks, it is doubtful that we gain much in terms of speed when compared to checking out things sequentially, asides from the fact that we don't even run the scheduler for these tasks)


Goals 11/21

@ebridge2

Status
Task
Deliverable

👍
Downloadable results
Deliverable: ability to download results from web service.

👍
Explorable QC
Deliverable: ability to explore QC integrated with the webpage.

👍
Errors when uploading data
Deliverable: demonstrate what happens when a duplicate subject is uploaded, then deleted, then uploaded, then analyzed (it throws an error now, make sure that doesn't happen by deliverable date).

👍
Overview of CompCorr
Deliverable: more rigorous overview of CompCorr (no code, just thorough pseudo for the algorithm and rough ideas for simulations; this week is mostly about reading and figuring out how to make it work with our pipeline). progress report

stretch goals

Status
Task
Deliverable

👍
Update pull request
Deliverable: pull request to ndmg.

@ewalke31 @02agarwalt

Status
Task
Deliverable

👍
Write pseudocode for KFS EM Discriminability code
Pseudocode

👍
Perform simulations for KFS EM discriminability
R markdown w/ simulations

:x:
If time, based on simulations, correct any errors we made with calculating discriminability on real data
R markdown w/ results (realdata.md)

@jovo @gkiar will these goals complete our deliverable of the web service itself for the final project (ie, having analysis and browsing of results for users?) just trying to gauge exactly what does/does not need to be in there for the final web service just so I can prioritize all of them in time.
also, @ewalke31 @02agarwalt add yo stuff.

@ebridge2
i don't want any choices in the pipeline, we choose for them.
if you want them to have multiple atlases, we will do all of them.
@ewalke31 we should have clarified more explicitly earlier, doc files are not an acceptable format, everything in this class must be web-readable.
@gkiar please update syllabus accordingly.

@ebridge2

[ ] upon downloading, what do you get? please clarify?
[ ] how do we know whether things look good?
[ ] use plot.ly + ggplot2 for figs
[ ] show the whole pipeline schematic
[ ] deliverables are figures/statistics

@ewalke31

[ ] what did we learn? simplify, look at it.
[ ] let's look at the A matrices?

@ewalke31 At the end of the "Perform simulations for KFS EM discriminability" document, you explain how we know that the estimation algorithm worked poorly. However, I wasn't clear on Why the estimation algorithm performed poorly.

@ebridge2  what port on cortex? couldn't see enough to get feedback from far away
also for the progress report, I find it neater to explain which symbol means what in a paragraph above the algorithm then just have the symbols as the input / output in the latex

@rmarren1 8061


fixing the error

Hi Daniel,
As you've highlighted in screenshot (https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/07-tensorboard-lowest-test-loss.png), the best model should be data_20_percent/effnetb0/10_epochs rather than effnetb2.

 Thank you for the correction!


Add Justlist: {JulyMoon72}

Please submit the correct information, invalid information will cause the issue to close


Admin endpoint to delete jobs.

Minimum proposal: A new admin endpoint that takes a job ID and deletes that job from the local DB and MinIO using the async platform's deleteJob method.  If the job is not owned by the campus the endpoint is called on, the endpoint should return some error indicating this.

something i realized: every release (2 months) we expire all compute jobs, to force recomputing with new data or new code.  that forced expiration will take care of these zombie jobs too.  so maybe not as urgent as i had thought..


Create a timeline

Add a list of all your activities, achievements, work experience. Add as much info as you can remember. In case you don’t have working experience tell about your pet-projects or student projects. Don’t be shy :)

Adding the timeline, with the next content:

I studied Computer Engineering at university, between 2017 and 2022

In December 2022 I started to do small personal projects, and I liked the frontend area with JS.

One of my future passions is to make mobile apps

Finish the timeline


Update info for slow commands

Per help from the slack channel, exporting the BUFFALO_PLUGIN_PATH env variable fixes the issue of slow command line performance.

@andy9775 What version of Buffalo do you use? Since v0.11.0 (https://github.com/gobuffalo/buffalo/commit/d024ca0b1ebcc6750aa93d6759f342db86b82e51), if this var is not set, it defaults to the $GOPATH/bin, and should be faster.

@stanislas-m I'm on 0.11.1.
Setting BUFFALO_PLUGIN_PATH did make it noticeably faster though.

@andy9775 Sorry for the (long) delay, and thanks!


Base64Url encode is not unique to KeyVault and should be in Azure Core

Base64Url encoding and decoding operation is a very common transformation API. It should be in Azure Core since we know future services will need it.
It is currently internal within KeyVault Common. If we are happy with the design, we should consider making it public and moving it to core.
https://github.com/Azure/azure-sdk-for-cpp/blob/0874e1c1c9047b9ccf6caac5057f67c9b92144a7/sdk/keyvault/azure-security-keyvault-common/inc/azure/keyvault/common/internal/base64url.hpp#L16-L22
https://github.com/Azure/azure-sdk-for-cpp/blob/0874e1c1c9047b9ccf6caac5057f67c9b92144a7/sdk/core/azure-core/inc/azure/core/base64.hpp#L15-L21
https://github.com/Azure/azure-sdk-for-net/search?q=base64url
WebPubSub, Attestation, ContainerRegistry, ACS, etc.
cc @JeffreyRichter, @RickWinter, @vhvb1989

As always making things public comes with a huge cost (making it general purpose [not just for our specific needs], documentation, samples, support in perpetuity). So, we should avoid this unless there is real customer benefit.

Leave it internal, but move it to core.


RFE: tmt test id option to check for duplicate IDs

The tmt CLI can be used to generate UUIDs for tests via tmt test id. Unfortunately, it does not check for duplicates. This can happen, for example, when tests are being copied and the ID is not adjusted - which can happen quite easily.
Therefore, I'd like to propose to add an additional check for duplication to the tmt test id command. Possible options I can think of are:

checking implicitly, generating new UUIDs for identified duplicates
provide another option to check and overwrite identified duplicates, e.g. --overwrite-duplicates
provide another option just to check for duplicates, e.g. --check-duplicates

For option 1. and 2. the --dry option would only print out the identified tests with duplicates and not change the file. But maybe there are other/better option? If this RFE is accepted and refined, I can also start working on it and provide a PR for it.
On a side note, it seems that tmt test id is missing in the documentation:
https://tmt.readthedocs.io/en/latest/stories/cli.html#stories-cli-test

Sounds also like a good idea for yet another tmt * lint check, one that looks not just at a single object at a time, but also keeps some notion of the global state, i.e. the existing IDs that were previously detected in the set of objects already examined.

I agree, I think it would fit good as another lint check.
And yes, if I understand it correctly (but only had a brief look at it), I'd need to introduce state into the tmt test id command (keeping track of the IDs). Is this acceptable or should this just be a lint check? It would be convenient to auto-fix duplicate IDs, though. Is there a tmt * lint --auto-fix or something similar? @happz

I agree, I think it would fit good as another lint check. And yes, if I understand it correctly (but only had a brief look at it), I'd need to introduce state into the tmt test id command (keeping track of the IDs). Is this acceptable or should this just be a lint check? It would be convenient to auto-fix duplicate IDs, though. Is there a tmt * lint --auto-fix or something similar? @happz

To me, this sounds more like an opportunity for tmt lint to be more useful. As you wrote at the beginning, this can happen when tests are copied and IDs are not modified - this is beyond what tmt test id can control because it has not been called to manage said IDs. I'm afraid we cannot prevent users from doing this kind of stuff, and tmt test id itself uses some magic to create unique IDs whenever it gets called, so the best option I see is to at least let the user know via linter there's a problem they brought on their heads :) Therefore I see no need for tmt test id to be concerned with anything but the given test.

It would be convenient to auto-fix duplicate IDs, though. Is there a tmt * lint --auto-fix or something similar? @happz

There is a --fix option, e.g. https://github.com/teemtee/tmt/blob/main/tmt/base.py#L1481 uses it, so we can assign a new ID to test if asked to do so. But while technically possible, I'm not sure if tmt should do it: let's say there are two tests with the same ID, which one of them is wrong? tmt creates IDs with the help of the uuid4() function, there is nothing related to the test reflected in the ID, so which one of those two tests should we fix? :)

There is a --fix option, e.g. https://github.com/teemtee/tmt/blob/main/tmt/base.py#L1481 uses it, so we can assign a new ID to test if asked to do so. But while technically possible, I'm not sure if tmt should do it: let's say there are two tests with the same ID, which one of them is wrong? tmt creates IDs with the help of the uuid4() function, there is nothing related to the test reflected in the ID, so which one of those two tests should we fix? :)

Initially, I assumed that it doesn't matter which test gets the new UUID assigned. Thinking about it, however, this is not true.
So the best solution is to extend tmt lint and list the tests containing any duplicate UUIDs found. Thanks for your help! @happz
I'd be up for implementing this issue :) Could you assign it to me?

We don't have a linter level that would focus on the repository, to consider objects as they are in it. We have tmt test lint which basically means "all tests", but can be executed for a subset too, tmt test lint /tests/foo/. So you will need some pre-lint step in code where a set of all IDs would be collected, then the check would be able to check this container and find out whether any of its IDs have more than one owner listed.
Or you can build the storage as you go, one by one, but then it would report no conflict if called for just one of the tests from that conflicting pair. It might be acceptable, I for one would not mind, because, eventually, I run tmt lint which sees everything in pre-commit, so I would learn about conflicts before committing.


Add deselect-last option

I found myself really wanting a way to deselect the last-selected selection even if the query has been cleared/changed, and couldn't for the life of me come up with any way to hack it together given the commands in place.
deselect-last simply deselects the last-selected selection, and repeated uses will pop back through the selections until there are none left, at which point it's a no-op.
Added the command, updated the docs, and added a test. Please just let me know if there's anything else I need to change here, or if anything doesn't look idiomatic (new to Go).
Local test run:
<glenn@ronnie:~/MrAwesome/fzf>
$ make
GOARCH=amd64 go build -a -ldflags "-s -w -X main.version=0.26.0 -X main.revision=3318bae" -tags "" -o target/fzf-linux_amd64
[Sat 21/04/03 21:39 CST][pts/11]
<glenn@ronnie:~/MrAwesome/fzf>
$ cp target/fzf-linux_amd64 bin/fzf
[Sat 21/04/03 21:39 CST][pts/11]
<glenn@ronnie:~/MrAwesome/fzf>
$ ruby test/test_go.rb --name test_deselect_last
Run options: --name test_deselect_last --seed 31866
<snip>
Finished in 0.172537s, 5.7959 runs/s, 63.7544 assertions/s.

1 runs, 11 assertions, 0 failures, 0 errors, 0 skips

Thanks. Please rename the variable to satisfy golint.

src/terminal.go:2401:6: don't use underscores in Go names; var len_selected should be lenSelected

By the way, can you give an example scenario where this would be really useful? I would personally use deselect-all if I want to undo the previous selections after changing the query, instead of "pop"ing them out one-by-one, especially because I probably wouldn't remember the selection order.

I'm working with this idea of using fzf as a sort of keyboard for Taiwanese Hokkien. Just started on it today so it's a bit rough, but here's the basic idea:

Pipe TW-US dictionary formatted a particular way into fzf, tab to select a chosen word and have it appear in the preview window as part of a sentence (without the definition), and have a live preview of the about-to-be-selected word at the end of the sentence you're constructing. Once you're happy with the sentence, mash enter and the indices of those chosen words are printed out for my original program to nom up and find the corresponding entries in the dict to get whatever information about those words I might need, parse/present them the way I want, and then send them off to X11 or what-have-you.
Just started on it today, but I'm really finding it to be a gigantic improvement over "go to online dictionary site, plug in one word at a time, wait for it to load, curse, repeat". fzf handles the latin letter normalization delightfully well, and searching for words is pretty easy. Got some work to do to figure out how to prioritize exact matches / shorter words, but beyond that it's working about as well as I could ever ask for.

Oh, got carried away and forgot to say the use case. I want a keyboard shortcut, plus backward-eof, to delete the last word in the sentence - otherwise you have to search a sometimes pretty precise phrase to find the last word, tab, then clear again before going on to the next word.

Oh, got carried away and forgot to say the use case. I want a keyboard shortcut, plus backward-eof, to delete the last word in the sentence - otherwise you have to search a sometimes pretty precise phrase to find the last word, tab, then clear again before going on to the next word.

Thinking about this some more, I am maybe trying to overload fzf's selection buffer by having it hold the state of my program. As clever as this feels, it does also feel a bit fragile. So it would maybe be best for me to just use execute-silent to send off selections elsewhere as they're made, and have preview just ask that file/process for the state of the built buffer.
I think it's best not to tie my program's functionality too deeply into a data structure I don't control that is intended for other uses. With that said, I'm happy to leave this up and push it through if anyone has a need/desire for this particular functionality. It did feel surprisingly intuitive to undo the last selections from anywhere (especially with backward-eof) instead of going back to each one in turn.

So are you still interested in pushing this to fzf? Or should we leave this open and see if anyone has similar requirements?
I'm not particularly against the idea, I just don't see myself using it in my workflow.


Added short explanation of exit codes.

Most importantly exit code -9 - which unless you know that it's a TIMEOUT_EXIT_CODE = -9 constant defined for runners, is very mysterious exit code.

LGTM.
I thought we also set timed_out attribute in the result object to True if the command times out, but that's actually not the case right now (we just set the error message to indicate the timeout and set status code to -9).

+1

Merged, thanks.


[HOTFIX] Remove unnecessary imports from #12944 that broke build

Test build #63241 has started for PR 14499 at commit a64ee0f.

Test build #3203 has started for PR 14499 at commit a64ee0f.

It has passed compilation, so I'm going to merge this in to fix the build.

Test build #3203 has finished for PR 14499 at commit a64ee0f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #63241 has finished for PR 14499 at commit a64ee0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63241/
Test PASSed.


Can't add iterable datasets to a Dataset Dict.

System Info
standard env
Who can help?
No response
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
Get the following error:
TypeError: Values in DatasetDict should be of type Dataset but got type '<class 'datasets.iterable_dataset.IterableDataset'>'
Expected behavior
should be able to add iterable datasets to a dataset dict

You need to use IterableDatasetDict instead of DatasetDict for iterable datasets.


Remove package tags

Description
This PR removes unnecessary @package tags ported from earlier CI versions.
According to the PHPDoc website:

The @package tag can be used as a counterpart or supplement to Namespaces. Namespaces provide a functional subdivision of Structural Elements where the @package tag can provide a logical subdivision in which way the elements can be grouped with a different hierarchy.
If, across the board, both logical and functional subdivisions are equal is it NOT RECOMMENDED to use the @package tag to prevent maintenance overhead.

In earlier CI versions the package tags were used as in place replacement for namespaces. Since CI4 is actively using namespaces, then usage of package tags is no longer necessary.
This PR also removes other superfluous tags ported from earlier CI versions.
Checklist:

[x] Securely signed commits
[x] Component(s) with PHPdocs

Thank you @paulbalandan

@paulbalandan there is @package left at BaseController in app directory, could you remove that as well? Thank you.


skim: changing number of columns to be summarized

skim summarizes 20 columns as default. I couldn't find to change this default behaviour.

For ease, I have removed the limit as a default. Please post a new issue if you'd like control over the number of columns that get summarised.


Health check initializingTimeout timer starts before container is running

The initializing timeout should only start once the container is running, or else it can expire before the image is even done downloading...
{
"id": "1i42",
"type": "container",
"links": {
"self": "…/v1/projects/1a5/containers/1i42",
"account": "…/v1/projects/1a5/containers/1i42/account",
"credentials": "…/v1/projects/1a5/containers/1i42/credentials",
"healthcheckInstanceHostMaps": "…/v1/projects/1a5/containers/1i42/healthcheckinstancehostmaps",
"hosts": "…/v1/projects/1a5/containers/1i42/hosts",
"instanceLabels": "…/v1/projects/1a5/containers/1i42/instancelabels",
"instanceLinks": "…/v1/projects/1a5/containers/1i42/instancelinks",
"instances": "…/v1/projects/1a5/containers/1i42/instances",
"mounts": "…/v1/projects/1a5/containers/1i42/mounts",
"ports": "…/v1/projects/1a5/containers/1i42/ports",
"serviceEvents": "…/v1/projects/1a5/containers/1i42/serviceevents",
"serviceExposeMaps": "…/v1/projects/1a5/containers/1i42/serviceexposemaps",
"services": "…/v1/projects/1a5/containers/1i42/services",
"targetInstanceLinks": "…/v1/projects/1a5/containers/1i42/targetinstancelinks",
"volumes": "…/v1/projects/1a5/containers/1i42/volumes",
"stats": "…/v1/projects/1a5/containers/1i42/stats",
"containerStats": "…/v1/projects/1a5/containers/1i42/containerstats",
},
"actions": {
"logs": "…/v1/projects/1a5/containers/1i42/?action=logs",
"setlabels": "…/v1/projects/1a5/containers/1i42/?action=setlabels",
},
"name": "Default_healthgood_1",
"state": "stopping",
"accountId": "1a5",
"build": null,
"capAdd": [ ],
"capDrop": [ ],
"command": null,
"cpuSet": null,
"cpuShares": null,
"createIndex": 1,
"created": "2016-03-17T19:33:01Z",
"createdTS":<PHONE_NUMBER>000,
"dataVolumeMounts": { },
"dataVolumes": [ ],
"dataVolumesFrom": [ ],
"dataVolumesFromLaunchConfigs": [ ],
"deploymentUnitUuid": "39ecc61a-2a6c-4162-8339-2ea761fd4789",
"description": null,
"devices": [ ],
"dns": [
"<IP_ADDRESS>",
],
"dnsSearch": [ 2 items
"default.rancher.internal",
"healthgood.default.rancher.internal",
],
"domainName": null,
"entryPoint": null,
"environment": null,
"expose": null,
"externalId": "8a4883c96d89a14a3f3a2425fe6e445ab52fc94b1dee0d64e4ea7b71d02b2119",
"extraHosts": null,
"firstRunning": null,
"healthCheck": {
"strategy": "recreate",
"interval": 2000,
"responseTimeout": 2000,
"initializingTimeout": 60000,
"healthyThreshold": 2,
"unhealthyThreshold": 3,
"requestLine": "GET \"/\" \"HTTP/1.1\\r\\nHost: blah.com\"",
"port": 80,
"name": null,
"reinitializingTimeout": null,
},
"healthState": "initializing",
"hostname": null,
"imageUuid": "docker:nginx",
"kind": "container",
"labels": {
"io.rancher.project.name": "Default",
"io.rancher.container.pull_image": "always",
"io.rancher.service.deployment.unit": "39ecc61a-2a6c-4162-8339-2ea761fd4789",
"io.rancher.service.launch.config": "io.rancher.service.primary.launch.config",
"io.rancher.project_service.name": "Default/healthgood",
"io.rancher.scheduler.affinity:container_label_soft": "io.rancher.service.deployment.unit=39ecc61a-2a6c-4162-8339-2ea761fd4789",
"io.rancher.stack.name": "Default",
"io.rancher.stack_service.name": "Default/healthgood",
},
"logConfig": null,
"lxcConf": null,
"memory": null,
"memorySwap": null,
"nativeContainer": false,
"networkContainerId": null,
"networkMode": "managed",
"pidMode": null,
"ports": [
"8888:80/tcp",
],
"primaryIpAddress": "<IP_ADDRESS>",
"privileged": false,
"publishAllPorts": false,
"readOnly": false,
"registryCredentialId": null,
"removed": null,
"requestedHostId": null,
"restartPolicy": null,
"securityOpt": null,
"startCount": 3,
"startOnCreate": true,
"stdinOpen": true,
"systemContainer": null,
"transitioning": "yes",
"transitioningMessage": "Post-network",
"transitioningProgress": 95,
"tty": true,
"user": null,
"uuid": "bc2b54cf-9deb-433d-9f6a-e0b68d32bd49",
"version": "0",
"volumeDriver": null,
"workingDir": null,
}

Tested with latest build from master:
Had a service with health check enabled and initializingTimeout set to 10000 ( 10 secs).
Provided incorrect value for "requestLine" parameter , so instance will be stuck in "Initializing" state forever.
@alena1108 , I see  that instances get removed after about 45 secs after it gets to "running" state.  I had set the  "initializingTimeout" to be 10 secs in this case . Is this delay in removal expected ?
+------+---------------------+---------------------+---------------------+---------+---------------------+--------------+---------------------+
| id   | name                | created             | removed             | state   | health_updated      | health_state | first_running       |
+------+---------------------+---------------------+---------------------+---------+---------------------+--------------+---------------------+
| 1701 | test1_testinitial_1 | 2016-03-20 01:40:08 | 2016-03-20 01:40:52 | purged  | 2016-03-20 01:40:09 | initializing | 2016-03-20 01:40:12 |
| 1702 | test1_testinitial_1 | 2016-03-20 01:41:08 | 2016-03-20 01:41:52 | purged  | 2016-03-20 01:41:08 | initializing | 2016-03-20 01:41:12 |
| 1703 | test1_testinitial_1 | 2016-03-20 01:41:52 | 2016-03-20 01:42:52 | purged  | 2016-03-20 01:41:53 | initializing | 2016-03-20 01:41:56 |
| 1704 | test1_testinitial_1 | 2016-03-20 01:43:08 | 2016-03-20 01:43:52 | purged  | 2016-03-20 01:43:08 | initializing | 2016-03-20 01:43:12 |
| 1705 | test1_testinitial_1 | 2016-03-20 01:43:52 | 2016-03-20 01:44:52 | purged  | 2016-03-20 01:43:53 | initializing | 2016-03-20 01:43:56 |
| 1706 | test1_testinitial_1 | 2016-03-20 01:44:52 | 2016-03-20 01:45:52 | purged  | 2016-03-20 01:44:53 | initializing | 2016-03-20 01:44:56 |
| 1707 | test1_testinitial_1 | 2016-03-20 01:45:52 | 2016-03-20 01:46:53 | purged  | 2016-03-20 01:45:54 | initializing | 2016-03-20 01:45:56 |
| 1708 | test1_testinitial_1 | 2016-03-20 01:46:52 | 2016-03-20 01:47:52 | purged  | 2016-03-20 01:46:54 | initializing | 2016-03-20 01:46:56 |
| 1709 | test1_testinitial_1 | 2016-03-20 01:47:52 | 2016-03-20 01:48:53 | purged  | 2016-03-20 01:47:53 | initializing | 2016-03-20 01:47:56 |
| 1710 | test1_testinitial_1 | 2016-03-20 01:49:08 | 2016-03-20 01:49:53 | removed | 2016-03-20 01:49:09 | initializing | 2016-03-20 01:49:12 |
| 1711 | test1_testinitial_1 | 2016-03-20 01:49:52 | 2016-03-20 01:50:53 | removed | 2016-03-20 01:49:54 | initializing | 2016-03-20 01:49:56 |
| 1712 | test1_testinitial_1 | 2016-03-20 01:50:52 | NULL                | running | 2016-03-20 01:50:54 | initializing | 2016-03-20 01:50:57 |
+------+---------------------+---------------------+---------------------+---------+---------------------+--------------+---------------------+
12 rows in set (0.01 sec)

| 694 | testinitial |          5 | service | c9320864-ba10-4b68-8896-a2d78c4dc114 | NULL        | active | 2016-03-20 01:40:05 | NULL    | NULL        | {"fields":{"scale":1,"secondaryLaunchConfigs":[],"launchConfig":{"kind":"container","networkMode":"managed","privileged":false,"publishAllPorts":false,"readOnly":false,"startOnCreate":true,"stdinOpen":true,"tty":true,"vcpu":1,"labels":{"io.rancher.container.pull_image":"always"},"imageUuid":"docker:sangeetha/testnewhostrouting","ports":[],"dataVolumes":[],"dataVolumesFrom":[],"dataVolumesFromLaunchConfigs":[],"dns":[],"dnsSearch":[],"capAdd":[],"capDrop":[],"devices":[],"healthCheck":{"strategy":"recreate","interval":2000,"responseTimeout":2000,"initializingTimeout":10000,"healthyThreshold":2,"unhealthyThreshold":3,"requestLine":"GET \"/name123.html\" \"HTTP/1.0\"","port":80,"name":null,"reinitializingTimeout":null},"count":null,"cpuSet":null,"cpuShares":null,"description":null,"domainName":null,"hostname":null,"memory":null,"memoryMb":null,"memorySwap":null,"pidMode":null,"requestedIpAddress":null,"user":null,"userdata":null,"volumeDriver":null,"workingDir":null,"networkLaunchConfig":null,"version":"0"},"startOnCreate":false,"assignServiceIpAddress":false,"token":"8qowkZsiK9hWXhcdztcFruhcjsbgkp7yBkBcm45y","transitioningProgress":50,"transitioningMessage":"Reconciling"}} |            336 | NULL |           19 | NULL          | NULL               | NULL        | initializing |
+-----+-------------+------------+---------+--------------------------------------+-------------+--

@sangeethah its ok, as cleanup is performed by periodic thread, so delays are expected


Keras 2.1.5: UnicodeDecode Error : 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

Hi
I  am trying to load a model from json and getting the UnicodeDecode error. I have reinstalled keras many times and replaced the generic_utils.py file even manually ( as I found some fixes on github ).
However, nothing works.
At first, I was trying to load an old model ( saved from keras 2.1.1 ) and it didnt work ( gave same unicodedecode errors ). Thus I re-run the network using keras 2.1.5 and saved new json model. However, the error remains.
@fchollet I even referred to https://github.com/keras-team/keras/issues/8738 and replaced the generic_utils.py but no change,
The error is as below:
kk@kk-VirtualBox:/media/sf_KK$ python test_network.py model_kk.h5
Using TensorFlow backend.
Traceback (most recent call last):
  File "test_network.py", line 101, in <module>
    model = model_from_json(jfile.read())
  File "/home/kk/miniconda3/envs/myenv/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

I am saving the model in the following way:
 model.save_weights('./model.h5')
    json_string = model.to_json()
    with open('./model.json', 'w') as f:
        f.write(json_string)

I am reading the model in the following way:
if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Test network')
    parser.add_argument('model', type=str,
    args = parser.parse_args()
    #args.model contains the model filename "model.h5"
    with open(args.model, 'r') as jfile:
           model = model_from_json(jfile.read())

Header of the model.h5 from vi
<89>HDF^M
^Z

>>> import keras
Using TensorFlow backend.
>>> print(keras.__version__)
2.1.5
>>>

[x] Check that you are up-to-date with the master branch of Keras. You can update with:
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

[x] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

[x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

[x] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

I reinstalled everything on different machine and was able to get rid of this error. not sure what caused it.