Feature: AMI430 driver

Driver for the American Magnetics 430 power supply programmer
To do:

[ ] Testing
[x] Separate drivers for 2D and 3D magnet operation

@alexcjohnson @giulioungaretti

@Rubenknex
I will post things which are still unclear to me.
So far, it's only small things.
I am missing a docstring for the meaning of the input arguments of the AMI430 driver init (and also for the 2D and 3D driver)
the AMI430._ramp_to command does not explicitely check any safety of the magnetic field if it is not part of a parent driver. It only passes the value trough the standard validator if you call the
        self.add_function('ramp_to',
                          call_cmd=self._ramp_to,
                          args=[Numbers(-self._field_rating,
                                        self._field_rating)])

function, but not if the user calls _ramp_to is called directly. So if the magnet is not part of a larger system, there is still a maximum field for this magnet, which needs to be checked by the _ramp_to function.

@Rubenknex
I found another way to quench the magnet:
If you only connect 1 magnet to QCoDes (in systems with only one magnet axis for example)
Then you can call _set_field with any input arguments which are not checked (even if perform_safety_check=True). So there another check for the value must be added. (I think this check should be performed regardles of the value of perform_safety_check)

@damazter Yes you're right, I added some checks.

@Rubenknex
I made a few inline comments, but more generally.
I am a bit confused by your different coordinate systems. It seems that If I set the field in spherical coordinates, the cartesian coordinates are not updated.
I think that it will go wrong if I am wanting to set something in cylindrical coordinates and first set rho, and after that start sweeping over phi, right?
so I think it is important that the different coordinate systems are bound together in some way.

@damazter I think I understand what you mean. I made a bunch of changes to how the coordinates are handled internally. Now all the setpoint coordinates are stored internally (self._x etc).
Before, when setting a coordinate, the current values were measured and then adjusted. This would result in an incremental error during a loop when for example the magnet does not go fully towards its setpoint.
Now, the coordinates stored internally are independent from the 'actual' magnet fields, and for every set they are all updated.
The code could be cleaner however, I could not think of an easy way to have all the coordinate parameters interact such that they all update when one is changed.

@Rubenknex
I am going through the code now, maybe I can come up with something
Maybe some renaming of the coordinates would be appropriate of field to 'r' for example
and if phi can be doubly used for spherical and cylindrical, then r/rho could be used for both as well

@Rubenknex
if I understand correctly the _update_internal_coords updates all internal coordinates, based on the value of _x, _y and _z. I guess that this function should then be executed after every _set_fields or am I mistaken?
Maybe the code could be more clear if you would explicitely separate the setpoint and the real field values. Then the real field values of the driver could only be x, y and z, fixed as parameters of the driver and only gettable.
And then make a different parameter for the setpoint which is gettable and settable. The problems with coordinate systems would only enter in the setpoint_parameters.
Putting 'setpoint' in th ename of the variable would make it a lot easier to read I think.
I am still thinking about the coordinate system problem. It would be tempting to put the entire x, y, z setpoint parameters as a single tuple parameter, and r, theta, phi setpoint parameters as a single, different parameter. But I don't know if qcodes supports tupled parameters? @giulioungaretti

@damazter I agree on separating the actual field and setpoints into separate parameters, that would also make it a lot clearer to the user. But then I guess you would want this for all the parameters (including spherical and cylindrical) right?
If we do this then I think the _update_internal_coords being called will depend on what the expected behavior for invalid setpoint will be (probably just keep the old one).
About the tuple parameters, I don't know how that would work with loops. Like would you select one value of the tuple to loop over?

@damazter Also another thing, currently in the 3D driver when changing phi and one of the magnets will be out of range, it will still set the others. Maybe it should just stop with an error and not do anything then?

Throwing an error earlier might be a good idea, so if you could make that, that would be great.
For the other post.
The actual field parameters only have to exist for x, y and z. I don't see an immediate reason to have all the others as well, because the user will be mainly interested in setting the setpoint and less with what the actual field is in terms of some different coordinate system. But I would not be opposed to adding the other coordinate systems to this.
For the setpoint it is a different matter and I think you will have to make a parameter for everyone of the nine coordinates.
tuples would be nicer, but I also don;t know about the loop integration of this. So we'd have to ask @giulioungaretti .
Another possibility might be to have the x1, x2 and x3 parameters in the driver, and then a single other parameter that defines the cooridnate system of the setpoint such as spherical, cylindrical, cartesian or even helical or weirder. That would make it more natural to distinguish between all the different systems.
In summary you would then have the parameters:
x
y
z
setpoint_r1
setpoint_r2
setpoint_r3
setpoint_coordinate_system (={CARtesian | SPherical | CYLindrical|...})
Then, upon changing the setpoint_coordinate_system you could change the units and labels of the coordinates r1, r2 and r3.
internally you might want to keep track of the setpoint in _x, _y and _z as well, but I think those shouldn't be QCodes parameters

@Rubenknex
Trying to set a field gives an error:
station.AMI430_3D.field(0.1)
yields
d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py:481: RuntimeWarning: invalid value encountered in double_scalars
  theta = np.arccos(z / field)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-d3a32f90175b> in <module>()
----> 1 station.AMI430_3D.field(0.1)

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in __call__(self, *args)
    133         else:
    134             if self.has_set:
--> 135                 self.set(*args)
    136             else:
    137                 raise NotImplementedError('no set cmd found in' +

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    869             e.args = e.args + (
    870                 'setting {} to {}'.format(self.full_name, repr(value)),)
--> 871             raise e
    872
    873     def _sweep_steps(self, value):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    861             clock = time.perf_counter()
    862             self.validate(value)
--> 863             self._set(value)
    864             self._save_val(value)
    865             if self._delay is not None:

d:\pycharmprojects\qcodes\qcodes\utils\command.py in __call__(self, *args)
    174             raise TypeError(
    175                 'command takes exactly {} args'.format(self.arg_count))
--> 176         return self.exec_function(*args)

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _set_field(self, field)
    586                                                                  self._field)
    587
--> 588         self._set_fields(self._x, self._y, self._z)
    589
    590     def _get_rho(self):

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _set_fields(self, x, y, z)
    628         # Finally, ramp up the coils that are increasing
    629         if not swept_x:
--> 630             self._magnet_x.field(x, perform_safety_check=False)
    631
    632         if not swept_y:

TypeError: ("__call__() got an unexpected keyword argument 'perform_safety_check'", 'setting AMI430_3D_field to 0.1')

@Rubenknex
I my opinion, this driver can be a bit more agressive when exceeding the field limits, or having other errors.
Now it will only print to the console of the ipython notebook, but raising an error might be appropiate. (Now, you don't see anything in the ipython notebook)

@Rubenknex
There should be an easier way to set x, y and z simultaniously. now, when I do
AMI.x(0.1)
AMI.y(0.1)
AMI.z(0.1)

I need to perform 9 sweeps. And even though 6 of them are trivial (they have the same start as ending point) they take a lot of time because of the sleeps in the _set_field function which is a waste of time.

@Rubenknex
I think I managed to quech the magnet again:
using multiprocessing (maybe that is cheating) you can start a sweep in x and y at the same time
all safety checks pass, because although the setpoints have been set, the actual field values are still zero.
so both magnet supplies are ramping to maximum at the same time.
this could be prevented by checking that none of the magnets is ramping when doing the field change. It is also possible that this not a scenario that is realistic. This is also only relevant if the AMI power control unit accepts multiple inputs which I don't know yet.

@Rubenknex
After you try to set an illegal value for one of the parameters you have to manually reset _x, _y or _z to be able to set the magnet again. This leads to another way to quench the magnet. By ramping x to 0.9 (with a maximal sphere of 1T) and then setting _x=0. Then calling AMI.y(0.9) will not see the security risk and ramp the y magnet.
I think this problem will be solved once all the stuff with the coordinate transformation and internal variables is solved. But just so you know what can lead to problems I have already included this in this discussion

@damazter Do you mean the logging.error calls don't show up in the notebook? Also raising an exception would probably stop your entire measurement (if you don't handle them), would this be wanted behavior?
The _set_fields function could probably check if the setpoints are identical to the last ones, preventing a pointless ramp. But maybe this is better to implement when we decide on how all the coordinate setpoints/current values should be represented. I can probably start implementing your idea of having separate parameters measuring (get only) and setpoints.

@damazter About the multiprocessing, it's probably a good idea to only have one magnet ramping at any time, which could be implemented easily I think by checking the state of the other two.
About the internal parameters, yes I think once we separate them into measured and setpoints, we don't have to store an internal state anymore.

@Rubenknex

@damazter Do you mean the logging.error calls don't show up in the notebook? Also raising an exception would probably stop your entire measurement (if you don't handle them), would this be wanted behavior?

The errors now don't show up in the notebook, which could be confusing. I am not opposed to just throwing an error. If you enter a wrong type (such as '0' instead of 0), an error is thrown, so throwing an error when the field is potentially exceded does feel natural to me.

The _set_fields function could probably check if the setpoints are identical to the last ones, preventing a pointless ramp. But maybe this is better to implement when we decide on how all the coordinate setpoints/current values should be represented. I can probably start implementing your idea of having separate parameters measuring (get only) and setpoints.

Sounds good!

About the internal parameters, yes I think once we separate them into measured and setpoints, we don't have to store an internal state anymore.

Same thing right? Again, sounds good!

@damazter About the multiprocessing, it's probably a good idea to only have one magnet ramping at any time, which could be implemented easily I think by checking the state of the other two.

Yeah, we indeed don't want two simultanious sweeps in the near future (and possible long term future), It would be super hard to prevent quenches hwen doing that, so I agree with your solution

@damazter I implemented most of the changes.

get-only *_measured parameters for returning a measured value
Internal state is stored in __x/y/z, this makes the parameters inaccessible. I could not store the setpoints the way I wanted using only the  Parameter class.
_set_fields is now responsible for performing all checks
If one or more magnets will exceed its limits, the call is aborted
New x/y/z values are checked for being equal with the np.isclose function, I don't know how accurate these magnets are, but by default it checks for differences larger than 1e-5 or 10 micro Tesla

The code has also become a bit cleaner.

@Rubenknex
I am not completely done with reading the driver, but some starting remarks

Internal state is stored in __x/y/z, this makes the parameters inaccessible. I could not store the setpoints the way I wanted using only the Parameter class.

This is not exactly true, the parameters are still accesable, but the names are just mangled. But I guess that should be enough

New x/y/z values are checked for being equal with the np.isclose function, I don't know how accurate these magnets are, but by default it checks for differences larger than 1e-5 or 10 micro Tesla

I think you should add quite some zero's there, just to be sure. And it should be possible to change this
Finally, I saw that you didn't go along with:

Another possibility might be to have the x1, x2 and x3 parameters in the driver, and then a single other parameter that defines the cooridnate system of the setpoint such as spherical, cylindrical, cartesian or even helical or weirder. That would make it more natural to distinguish between all the different systems.
In summary you would then have the parameters:
x
y
z
setpoint_r1
setpoint_r2
setpoint_r3
setpoint_coordinate_system (={CARtesian | SPherical | CYLindrical|...})
Then, upon changing the setpoint_coordinate_system you could change the units and labels of the coordinates r1, r2 and r3.
internally you might want to keep track of the setpoint in _x, _y and _z as well, but I think those shouldn't be QCodes parameters

I personally think this might be easier than the system that is in place now, any thoughts?

@damazter
Ah ok, I'm not sure about a  better solution, python is not good at hiding stuff from users haha.
I could add a parameter for the setpoint checking precision.
About your idea with reusing the parameters, I'm not sure which approach is better, but here are a few things:

Say a user changes his coordinate system somewhere without setting it every time he sets a parameter. Now he might set an 'x' with an angle or something without this being immediately obvious.
From the code it will not be very clear what is happening when there is setpoint_r2(0.5)
I personally prefer an interface (what the parameters actually do) that is static instead of dynamic if it is not absolutely neccesary (for some intruments the state matters a lot, but here we can still do everything simultaniously)

But let me know what you think of this

@Rubenknex
You're right, hiding stuff is hard
parameters for precision checking would work
you know about ManualParameter which implements that?
Also I see that the _measure function only returns a single value. This is quite a waste of time if you wan't to know all three, isn't it?
line 593:
    def _set_field(self, field):
        phi, theta, field, rho = self._get_non_cartesian()

        field = value

value not defined (i think the argument in the def should be changed form field to value)
The biggest problem I can think of now for not having the r1, r2, r3 method is when sweeping, you will accumulate rounding errors when sweeping in spherical coordinates because you transform back and forth to cartesian instead of only going in one direction.
Implementing other coordinate systems is then also difficult, but i guess they should interface directly to _set_fields Maybe it would make sense making that method nonprivate.
Also, say you want to sweep two spherical coordinates at the same time, you can only make subsequent calls to change them. This means that the sweepingroute is far from optimal (take the case where in the simultaneous case, one of the cartesian coordinates is constant). A solution would be is that all methods in the driver only update the setpoint and that it will start ramping when you call some extra ramp function which then does the actual ramping of the field.
Then with the r1, r2, r3 method, r1, r2 and r3 would just be manual parameters. This would make the code of the driver quite a lot shorter. I would even be in favor of making a single parameter setpoint that contains the 3-dimensional setpoint. Right now, the loop cannot really deal with that, but it will have to in the future so it wouldn't be that weird to already implement it in this driver. (Sorry @giulioungaretti )
If we stick with the current implementation, it would be sort of similar to what it now is, but it would be possible to have a more optimal path when changing two parameters simultaneously (without ramping two direction at the same time)

@damazter
I also thought about the rounding errors, but a simple (not rigorous) test shows that sweeping phi over 360 degrees in 10000 steps gives only an error on the order of 1e-13 compared to the starting x/y/z components. Maybe a better test is needed, but I don't think this is a problem.
That's a nice idea, first setting setpoints and then ramping, it would make the code a lot cleaner and indeed solve most of the problems until now. One thing I can think of right now; how would one do a loop? You cannot do Loop(magnet.phi[0, 180, 10]) anymore.

@damazter @giulioungaretti
Tuple parameters have been added to the driver in the form of
get() -> (x,y,z)
set((x,y,z))
for all coordinate systems. The code also became a bit cleaner.

@Rubenknex
Two quick notes,
on line 616 there is a line field = value but value is not defined
secondly the np.isclose still needs to be changed such that it becomes more predictable what close means. A safe bet would be to set no relative_tolerance but only an absolute tolerance of 10^-8

everytime the instrument is initialized I get the error:
qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py:540: RuntimeWarning: invalid value encountered in double_scalars
  theta = np.arccos(z / field)

So I think it would make more sense to define theta to be 0 or something when there is no field

The validator on the cartesian parameter makes it impossible to use the parameter, because the vector that you supply to the cartesian parameter is not a number:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-e5ddc0628e4a> in <module>()
----> 1 station.AMI430_3D.cartesian((0.5,0,0))

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in __call__(self, *args)
    133         else:
    134             if self.has_set:
--> 135                 self.set(*args)
    136             else:
    137                 raise NotImplementedError('no set cmd found in' +

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    868             e.args = e.args + (
    869                 'setting {} to {}'.format(self.full_name, repr(value)),)
--> 870             raise e
    871
    872     def _sweep_steps(self, value):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    859         try:
    860             clock = time.perf_counter()
--> 861             self.validate(value)
    862             self._set(value)
    863             self._save_val(value)

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in validate(self, value)
    335             context = self.name
    336
--> 337         self._vals.validate(value, 'Parameter: ' + context)
    338
    339     def sweep(self, start, stop, step=None, num=None):

d:\pycharmprojects\qcodes\qcodes\utils\validators.py in validate(self, value, context)
    158         if not isinstance(value, self.validtypes):
    159             raise TypeError(
--> 160                 '{} is not an int or float; {}'.format(repr(value), context))
    161
    162         if not (self._min_value <= value <= self._max_value):

TypeError: ('(0.5, 0, 0) is not an int or float; Parameter: AMI430_3D.cartesian', 'setting AMI430_3D_cartesian to (0.5, 0, 0)')

The same holds for cylindrical and spherical

Unfortunately the default validator is Numbers, so I guess you have to set it to the any validator to circumvent the validator all together

I am having a new problem and although I think I know what caused it, I am not fully sure how to solve it: The return value af ask('QU?') is parsed as an int but checked against string values.
Would it be possible to preparse everything to either a string or an int just to be sure what we are getting? (the same is true for the other parameters i guess)
It is strange that I have not seen this error before, I am not sure why that wouyld be.
Note that I am running agains a dummy version of the instrument, so it is a little hard to check that my dummy version works perfectly, but I think it would be good to build stability for str vs int anyway
error:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _valmapping_get_parser(self, val)
    831             val = int(val)
--> 832             return self._get_mapping[val]
    833         except (ValueError, KeyError):

KeyError: 0

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-5-e5ddc0628e4a> in <module>()
----> 1 station.AMI430_3D.cartesian((0.5,0,0))

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in __call__(self, *args)
    133         else:
    134             if self.has_set:
--> 135                 self.set(*args)
    136             else:
    137                 raise NotImplementedError('no set cmd found in' +

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    868             e.args = e.args + (
    869                 'setting {} to {}'.format(self.full_name, repr(value)),)
--> 870             raise e
    871
    872     def _sweep_steps(self, value):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _validate_and_set(self, value)
    860             clock = time.perf_counter()
    861             self.validate(value)
--> 862             self._set(value)
    863             self._save_val(value)
    864             if self._delay is not None:

d:\pycharmprojects\qcodes\qcodes\utils\command.py in __call__(self, *args)
    175             raise TypeError(
    176                 'command takes exactly {} args'.format(self.arg_count))
--> 177         return self.exec_function(*args)

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _set_fields(self, values)
    668             swept_x = True
    669         elif np.abs(self._magnet_x.field()) < np.abs(x):
--> 670             self._magnet_x._set_field(x, perform_safety_check=False)
    671             swept_x = True
    672

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _set_field(self, value, perform_safety_check)
    177             return
    178
--> 179         if self._can_start_ramping():
    180             self.pause()
    181

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _can_start_ramping(self)
    136         Check the current state of the magnet to see if we can start ramping
    137         """
--> 138         if self.is_quenched():
    139             logging.error(__name__ + ': Could not ramp because of quench')
    140             return False

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in __call__(self, *args)
    127         if len(args) == 0:
    128             if self.has_get:
--> 129                 return self.get()
    130             else:
    131                 raise NotImplementedError('no get cmd found in' +

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in get(self)
    809         except Exception as e:
    810             e.args = e.args + ('getting {}'.format(self.full_name),)
--> 811             raise e
    812
    813     def _valmapping_get_parser(self, val):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in get(self)
    804     def get(self):
    805         try:
--> 806             value = self._get()
    807             self._save_val(value)
    808             return value

d:\pycharmprojects\qcodes\qcodes\utils\command.py in __call__(self, *args)
    175             raise TypeError(
    176                 'command takes exactly {} args'.format(self.arg_count))
--> 177         return self.exec_function(*args)

d:\pycharmprojects\qcodes\qcodes\utils\command.py in call_by_str_parsed_out(self, *args)
    128     def call_by_str_parsed_out(self, *args):
    129         """Execute a formatted string with output parsing."""
--> 130         return self.output_parser(self.exec_str(self.cmd_str.format(*args)))
    131
    132     def call_by_str_parsed_in(self, arg):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in _valmapping_get_parser(self, val)
    832             return self._get_mapping[val]
    833         except (ValueError, KeyError):
--> 834             raise KeyError('Unmapped value from instrument: {!r}'.format(val))
    835
    836     def _valmapping_with_preparser(self, val):

KeyError: ('Unmapped value from instrument: 0', 'getting AMI430_X_is_quenched', 'setting AMI430_3D_cartesian to (0.5, 0, 0)')

@damazter
I am a bit confused about why this error occurs. The code in parameter.py looks like:
# Try and match the raw value from the instrument directly
        try:
            return self._get_mapping[val]
        except KeyError:
            pass

        # If there is no match, we can try to convert the parameter into a
        # numeric value
        try:
            val = int(val)
            return self._get_mapping[val]
        except (ValueError, KeyError):
            raise KeyError("Unmapped value from instrument: {!r}".format(val))

So apparently it's failing to do self._get_mapping[val] where self._get_mapping={'0':False, '1':True}. This is strange because '0' is being returned by the instrument right?
How are you testing it as a dummy instrument by the way?

@Rubenknex
My dummy instrument connects over TCPIP, so that is the same as the instrument intself. The instrument returns '0\r\n' but apperantly that ends up as being 0 in the driver. Wouldn't it be easiest to convert everything to a string before doing the value mapping. (Is there an option to do something like that in qcodes parameters?)

@damazter
Can you check if changing the terminator='\n' to '\r\n' fixes it? Maybe your '0\r\n' is being stripped by qcodes to '0\r' which then cannot be found in the dict.
Does the magnet return it's messages with \r\n? If so and the above fixes it then I will change the terminator to that.

@Rubenknex Indeed that sovled the problem. I changed the line terminator before, but I accidently overwrote it when pulling the new version of your driver. Stupid me.
would it be reasonable to check for this? But maybe this is also up to the user, to configure this correctly.
I think it would make sense to change the behaviour of the driver a little bit though. If terminator is given as a kwarg to the constructor, I think this given terminator should be used in favor of '\n' (which is not the case now, as you overwrite it when calling super().__init))

@damazter
Yes I think it's a good idea for the user to be able to change it, I updated it in the latest commit.

@Rubenknex
getting there, but I am still finding some errors:
I think the problem is somewhere in the partial function, but I am not sure
executing station.AMI430_3D.field()
gives
KeyError                                  Traceback (most recent call last)
<ipython-input-6-1e06fb7e6062> in <module>()
----> 1 station.AMI430_3D.field()

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in __call__(self, *args)
    127         if len(args) == 0:
    128             if self.has_get:
--> 129                 return self.get()
    130             else:
    131                 raise NotImplementedError('no get cmd found in' +

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in get(self)
    809         except Exception as e:
    810             e.args = e.args + ('getting {}'.format(self.full_name),)
--> 811             raise e
    812
    813     def _valmapping_get_parser(self, val):

d:\pycharmprojects\qcodes\qcodes\instrument\parameter.py in get(self)
    804     def get(self):
    805         try:
--> 806             value = self._get()
    807             self._save_val(value)
    808             return value

d:\pycharmprojects\qcodes\qcodes\utils\command.py in __call__(self, *args)
    175             raise TypeError(
    176                 'command takes exactly {} args'.format(self.arg_count))
--> 177         return self.exec_function(*args)

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _get_setpoints(self, *names)
    578
    579     def _get_setpoints(self, *names):
--> 580         return self._from_xyz(self.__x, self.__y, self.__z, names)
    581
    582     def _set_x(self, value):

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in _from_xyz(self, x, y, z, *names)
    563         }
    564
--> 565         returned = tuple(coords[name] for name in names)
    566
    567         if len(returned) == 1:

d:\pycharmprojects\qcodes\qcodes\instrument_drivers\american_magnetics\AMI430.py in <genexpr>(.0)
    563         }
    564
--> 565         returned = tuple(coords[name] for name in names)
    566
    567         if len(returned) == 1:

KeyError: (('field',), 'getting AMI430_3D_field')

@damazter
I made a small type, it should work now.

@Rubenknex
Thanks, that works now.
I found however that when initializing the driver, the setpoints are set to zero, but the magnet might not be. This leads to very strange behaviour as it becomes impossible to actually set the magnet to zero because of
        if np.isclose(self.__x, x, rtol=0, atol=1e-8):
            swept_x = True

etc.
I think this can be solved by setting __x, __y and __z to the meaured field values at the initialization.
Problems will still arise when the magnet does stuff which the driver is unaware off, but I don't know if that can happen.

@damazter
That's an interesting point, the values are now initialized and I added a method update_internal_setpoints that can be called when stuff has been changed manually.

@Rubenknex
nice
You forgot to define the magnets though:
    def update_internal_setpoints(self):
        """
        Set the internal setpoints to the measured field values.
        This can be done in case the magnets have been adjusted manually.
        """
        self.__x = magnet_x.field()
        self.__y = magnet_y.field()
        self.__z = magnet_z.field()

@Rubenknex
could it be that this line of code is the wrong way around?
(similarly for y and z)
        elif np.abs(self._magnet_x.field()) < np.abs(x):

(line: 697)

@damazter
Ah that is indeed the case, I think I mixed up 'less than' and 'decreasing' or something.

@Rubenknex
can I propose to change
phi, theta, field = values
into
field, theta, phi = values
First of all, it would be nice to have a right-handed coordinate system.  and I think it is most natural to have r as the first coordinate.
Then I think the natural extension to cylindrical coordinates is
rho, phi, z = values
instead of
phi, rho, z = values

@Rubenknex
We had some problem with the order of the coordinates in spherical coordinates. I think that the order is different for setting and getting the spherical magnetic field, right?

@damazter Yes there were some unchanged lines from your last comment about changing the order. It should be fixed now.

I suggest to, merge this. And open a new PR with things missing, fixes and so on!
It's a big PR with many comments, and possibly really hard to read in the future!
Let me know what you think!

@giulioungaretti
I would be all for it,
@Rubenknex
I noticed that the ramprates are not set in the init of this driver. I think it would be wise to check if the ramprate is within the safe limits however, such that you are not accidnetly stuck with a very wrong ramping speed.

@Rubenknex merge and open new ?

@giulioungaretti Yes that's fine by me, this PR is getting quite long haha


Login page + client-server authentication

Let's leave it for now, and if it becomes an issue we can just .gitignore it


Issue #936: Refactor Unit Test

Results from unittest.exe -
Execution finished in 32.298 seconds. Failed: 0 Passed: 56 Skipped: 105 Percentage: 100.00%
[ Please outline your change here. Be sure to check the checkboxes below. ]
Code change checklist

[x] I have ensured that all methods and functions are fully documented using doxygen style comments.
[x] My code follows the coding style guide.
[x] I tested that my change works before raising the PR.
[x] I have ensured that I did not break any existing API calls.
[x] I have not built my pull request using AI, a static analysis tool or similar without any human oversight.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

105 tests were skipped, does this still work with online tests?

unit test with token -
UNIT TEST SUMMARY
==================
dpp::snowflake class                                         PASS
dpp::json_interface class                                    PASS
Instantiate DPP cluster                                      FAIL
cluster::start method                                        NOT EXECUTED
Connection to client websocket                               FAIL
Creation of application command                              NOT EXECUTED
Deletion of application command                              NOT EXECUTED
Log events                                                   NOT EXECUTED
Creation of a channel message                                NOT EXECUTED
Editing a channel message                                    NOT EXECUTED
Message edit event                                           NOT EXECUTED
Deletion of a channel message                                NOT EXECUTED
Receipt of a created message                                 NOT EXECUTED
Message attachment send and check                            NOT EXECUTED
Test guild cache                                             NOT EXECUTED
Test user cache                                              NOT EXECUTED
React to a message                                           NOT EXECUTED
Reaction event                                               NOT EXECUTED
Receive guild create event                                   FAIL
Get messages                                                 NOT EXECUTED
crossplatform_strptime()                                     NOT EXECUTED
utility::iconhash                                            PASS
cluster::current_user_get()                                  NOT EXECUTED
cluster::guild_get()                                         NOT EXECUTED
cluster::channel_get()                                       NOT EXECUTED
cluster::channels_get()                                      NOT EXECUTED
cluster::roles_get()                                         NOT EXECUTED
cluster::guild_get_invites()                                 NOT EXECUTED
cluster::guild_get_bans()                                    NOT EXECUTED
cluster::channel_pins_get()                                  NOT EXECUTED
cluster::guild_events_get()                                  NOT EXECUTED
cluster::guild_event_get()                                   NOT EXECUTED
message_create_t::send()                                     NOT EXECUTED
cluster::guild_event_users_get()                             NOT EXECUTED
start timer                                                  NOT EXECUTED
stop timer                                                   NOT EXECUTED
one-shot timer                                               NOT EXECUTED
timed listener                                               NOT EXECUTED
Presence intent                                              FAIL
Instantiate a cache                                          PASS
message_collector                                            FAIL
managed::get_creation_date()                                 PASS
utility::read_file()                                         PASS
ts_to_string()                                               PASS
ts_not_null()                                                PASS
command_option_choice::fill_from_json: double                PASS
command_option_choice::fill_from_json: int64_t               PASS
command_option_choice::fill_from_json: bool                  PASS
command_option_choice::fill_from_json: snowflake             PASS
command_option_choice::fill_from_json: string                PASS
https_client::get_host_info()                                PASS
https_client HTTPS request                                   FAIL
https_client HTTP request                                    PASS
run_once<T>                                                  PASS
webhook construct from URL                                   PASS
Markdown escaping (ignore code block contents)               PASS
Markdown escaping (escape code block contents)               PASS
URL encoding                                                 PASS
Base 64 encoding                                             PASS
sync<T>()                                                    FAIL
manged object comparison                                     PASS
find_channel()                                               NOT EXECUTED
channel type flags                                           NOT EXECUTED
create a forum channel                                       NOT EXECUTED
retrieve the created forum channel                           NOT EXECUTED
delete the created forum channel                             NOT EXECUTED
Human readable error translation                             PASS
cluster::guild_ban_add ban three deleted discord accounts    NOT EXECUTED
cluster::guild_get_ban getting one of the banned accounts    NOT EXECUTED
cluster::guild_get_bans get bans using the after-parameter   NOT EXECUTED
cluster::guild_ban_delete unban the banned discord accounts  NOT EXECUTED
cluster::thread_create                                       NOT EXECUTED
cluster::on_thread_create event                              NOT EXECUTED
cluster::channel_delete with thread                          NOT EXECUTED
cluster::on_thread_delete event                              NOT EXECUTED
cluster::thread_edit                                         NOT EXECUTED
cluster::on_thread_update event                              NOT EXECUTED
cluster::threads_get_active                                  NOT EXECUTED
creating a voice channel                                     NOT EXECUTED
editing the created voice channel                            NOT EXECUTED
deleting the created voice channel                           NOT EXECUTED
permission                                                   PASS
event class                                                  PASS
cluster::user_get                                            NOT EXECUTED
cluster::user_get flag parsing                               NOT EXECUTED
cluster::guild_get_member                                    NOT EXECUTED
user::get_mention                                            PASS
user::get_url                                                PASS
user::format_username                                        PASS
user::get_creation_time                                      PASS
user::get_avatar_url                                         PASS
channel::set_type                                            PASS
channel::get_mention                                         PASS
channel::get_url                                             PASS
message::get_url                                             PASS
utility::guild_navigation                                    PASS
utility::iconhash                                            PASS
utility::make_url_parameters                                 PASS
utility::markdown_escape                                     PASS
utility::tokenize                                            PASS
utility::url_encode                                          PASS
utility::slashcommand_mention                                PASS
utility::channel_mention                                     PASS
utility::user_mention                                        PASS
utility::role_mention                                        PASS
utility::emoji_mention                                       PASS
utility::user_url                                            PASS
utility::message_url                                         PASS
utility::channel_url                                         PASS
utility::thread_url                                          PASS
utility::avatar_size                                         PASS
utility::cdn_endpoint_url_hash                               PASS
sticker::get_url aka utility::cdn_endpoint_url_sticker       PASS
emoji::get_url                                               PASS
role::operator<                                              PASS
cluster::role_create                                         NOT EXECUTED
cluster::role_edit                                           NOT EXECUTED
cluster::role_delete                                         NOT EXECUTED
JSON parse error for post_rest                               PASS
cluster::user_get_cached_sync() with present member          FAIL
cluster::user_get_cached_sync() with not present member      PASS
interaction_create_t::get_parameter() with subcommands       PASS
interaction_create_t::get_parameter() without subcommands    PASS
cluster::automod_rule_create                                 NOT EXECUTED
cluster::automod_rule_get                                    NOT EXECUTED
cluster::automod_rules_get                                   NOT EXECUTED
cluster::automod_rule_delete                                 NOT EXECUTED
using the cluster::request method to fetch an image          NOT EXECUTED
cluster::guild_emoji_create                                  NOT EXECUTED
cluster::guild_emoji_get                                     NOT EXECUTED
cluster::guild_emoji_delete                                  NOT EXECUTED
cluster::on_invite_create                                    NOT EXECUTED
cluster::on_invite_delete                                    NOT EXECUTED
cluster::channel_invite_create                               NOT EXECUTED
cluster::invite_get                                          NOT EXECUTED
cluster::invite_delete                                       NOT EXECUTED
multiheader cookie test                                      SKIPPED
Connect to voice channel                                     SKIPPED
Send audio to voice channel                                  SKIPPED
Pinning a channel message                                    SKIPPED
Unpinning a channel message                                  SKIPPED
cluster::thread_member_add                                   SKIPPED
cluster::thread_member_get                                   SKIPPED
cluster::thread_members_get                                  SKIPPED
cluster::thread_member_remove                                SKIPPED
cluster::on_thread_members_update event with member addition SKIPPED
cluster::on_thread_members_update event with member removal  SKIPPED
cluster::thread_create_with_message                          SKIPPED
message manipulation in thread                               SKIPPED
cluster::on_message_create in thread                         SKIPPED
cluster::on_message_edit in thread                           SKIPPED
cluster::on_message_delete in thread                         SKIPPED
cluster::on_reaction_add in thread                           SKIPPED
cluster::on_reaction_remove in thread                        SKIPPED
coro: offline job                                            SKIPPED
coro: offline coroutine                                      SKIPPED
coro: offline task                                           SKIPPED
coro: offline async                                          SKIPPED
coro: online event handler                                   SKIPPED
coro: online api calls                                       SKIPPED
coro: online mumbo jumbo in event handler                    SKIPPED

Execution finished in 3.441 seconds.
Failed: 78 Passed: 58 Skipped: 25 Percentage: 42.65%

Thanks for contributing!
Make sure you run the full suite of tests including online tests. You'll need to set a couple environment variables, see https://dpp.dev/unit-tests.html
Also pass "full" as the first command argument for some extra tests

Thanks for contributing!
Make sure you run the full suite of tests including online tests. You'll need to set a couple environment variables, see https://dpp.dev/unit-tests.html
Also pass "full" as the first command argument for some extra tests

Thanks for the suggestion, I was looking for environment variables details.

fails on mac os:
https://github.com/brainboxdotcc/DPP/actions/runs/7863356947/job/21453913274?pr=1080

Hi there, are you currently working on readding mac support and fixing the other issues? You seemed to fix the space indentation but left everything else.

We've closed this PR due to no activity. The PR also seems to be a direct copy of an old PR that was raised, with some additions. This also breaks our OSX support.


Adding beef stew recipe

Traditional Irish stew with beef or lamb.

Add a link to the page in index.md too, like the others.

Done. Thanks Luke.


the loss is nan

Hi,
when i run python train.py on coco data, follow your tutorial exactly:
i get :
=> STEP   49   lr: 0.000020   giou_loss: 9.46    conf_loss: 42.83   prob_loss: 6.38   total_loss: 58.67
=> STEP   50   lr: 0.000020   giou_loss:  nan     conf_loss:  nan     prob_loss:  nan   total_loss:  nan
=> STEP   51   lr: 0.000021    giou_loss:  nan      conf_loss:  nan     prob_loss:  nan   total_loss:  nan
=> STEP   52   lr: 0.000021    giou_loss:  nan     conf_loss:  nan     prob_loss:  nan   total_loss:  nan
All parameters I use are default。
in addition， I think the nine line of main function in train.py should be ：
steps_per_epoch = len(trainset) // cfg.TRAIN.BATCH_SIZE，
Your code is：
steps_per_epoch = len(trainset)
I am a beginner of computer vision, I don't know if my understanding is wrong ？

We noticing the issue too, it is because of use of softplus without threshold. It'll updated soon.

Hi
When I run train.py code on my custom data, I get nan value too.
Did you solve this problem now? I want to solution this problem, but I'm beginner of CV too...
I think my dataset is a problem...
This is a example my train set & test set.  this is a text file.
img_path x_min,y_min,x_max,y_max,label_number
ex)
./image/train/Edwin.png 153,21,300,400,0
is this alright to use dataset for training??
I need your help... Thanks you.
(Sorry for my English skill is worst.. I need you understand What I'm talking about..)

I had same problem, solved it by changing learning rate to 1e-4 and using my own way for transfer learning (here)

I had same problem, solved it by changing learning rate to 1e-4 and using my own way for transfer learning (here)

Thank you for your help! I solve this problem. Because of .weight file, I can't learning my own dataset. So I don't use yolov4.weight file. Just make a new .weights files. Thank you!

I had same problem, solved it by changing learning rate to 1e-4 and using my own way for transfer learning (here)

which learning rate is set to 1e-4 ? lr_init or lr_end ?
and even I convert weight files from .weights to .pkl, my loss is also non on coco dataset

We're noticing the issue too, it is due to use of softplus without threshold. It'll be updated soon.

please let me know if you can do this, thanks

I had same problem

i found in yolov4.py use the sigmoid.

Same problem，too.

me too, Same problems....

me too, Same problems....

I replaced the mish activation with leaky_relu in common.py and it worked for me

thank you very much
------------------ Original ------------------
From: zateyev<EMAIL_ADDRESS>Date: Thu,Jun 4,2020 0:21 PM
To: hunglc007/tensorflow-yolov4-tflite<EMAIL_ADDRESS>Cc: Demo Xu<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [hunglc007/tensorflow-yolov4-tflite] the loss is nan (#14)
I replaced the mish activation with leaky_relu in common.py and it worked for me
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

I replaced the mish activation with leaky_relu in common.py and it worked for me

When training to more than one thousand iterations, the loss value is still Nan....

The 'bbox_iou' function on line 177 in 'yolov4.py' is wrong, This may be related to the loss of the network as nan.
https://github.com/hunglc007/tensorflow-yolov4-tflite/issues/24

I replaced the mish activation with leaky_relu in common.py and it worked for me

I use this method to train my yolov4, the loss is normal, but the ap is lower compared with using mish as the activation function.

The 'bbox_iou' function on line 177 in 'yolov4.py' is wrong, This may be related to the loss of the network as nan.
#24

so , How to modify ?

@zcdliuwei
You need to modify：
left_up = tf.maximum(boxes1_coor[..., :2], boxes1_coor[..., :2])
to：
left_up = tf.maximum(boxes1_coor[..., :2], boxes2_coor[..., :2])

@zxj11838  are you serious? how about  right_down = tf.minimum(boxes2_coor[..., 2:], boxes2_coor[..., 2:])

According to my understanding, here is the calculation of the coordinates of the upper left corner and lower right corner of the intersection of the two boxes.

I modified the following content according to the readme，the loss of training and testing is not nan.
Traning your own model
Prepare your dataset
If you want to train from scratch:
In config.py set FISRT_STAGE_EPOCHS=0
Run script:
python train.py

I modified the following content according to the readme，the loss of training and testing is not nan.
Prepare your dataset
If you want to train from scratch:
In config.py set FISRT_STAGE_EPOCHS=0

I think this warehouse is the best  implemented in Tensorflow2.X of yolov4 so far, but the problem that the loss value is Nan has not been solved,  I've tried to modify some things, such as training from scratch, modifying activation functions, etc，but they all didn't work,  now I don't know what to do anymore, because my loss value has always been Nan, can you send me a copy of your code ?

Same problem here, even after using leaky_relu, loss will be nan after some steps.
Can't be use for training.

I want to ask for your advice. I want to identify the bottles in the trash can. The real image is about this:

The most complex part of this scene is the dense shielding and stacking of bottles. Our purpose is to tell the customer who threw the bottle what kind of bottle he threw in, and it's better to return the location information of the bottle at the bottom of the trash can. At present, I use Yolo series for testing, and the effect is always unsatisfactory.
I want to know does you have any suggestions to improve the detection and classification ability in this scenario?
Much obliged for your assistance.

I have the same problem with the git repo downloaded and installed last week. I tried the suggestion by @bgamini , but that did not help, perhaps because I am training my own data from scratch. I am giving more details of my case, and would really appreciate your suggestions.
My environment:
Python 3.7 /  NVIDIA GeForce RTX 3070/ cudart64_101.dll/ cudnn64_7.dll/
Other libraries according to according to requirements-gpu.txt
I am trying to train my custom data (with 3 claases) from scratch (without using yolov4.weights file). I prepared the dataset with the required annotation files, following the example of voc_annotation.  I also updated the contents and paths of classes.names,and  train.txt / test.txt  in ./data/classes and ./data/dataset folders respectively.
In addition,
I updated  __C.TRAIN.FISRT_STAGE_EPOCHS    = 0 in config, and
updated FLAGS.weights as
flags.DEFINE_string('weights', None, 'pretrained weights')
=> TEST STEP   36   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan
<core.dataset.Dataset object at 0x000002388F00FB70> 5
=> STEP   36/ 150   lr: 0.000917   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan
=> STEP   37/ 150   lr: 0.000911   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan
=> STEP   38/ 150   lr: 0.000905   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan
=> STEP   39/ 150   lr: 0.000898   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan
=> STEP   40/ 150   lr: 0.000891   giou_loss:  nan   conf_loss:  nan   prob_loss:  nan   total_loss:  nan

I forgot to add that this  error shows up only with GPU and not if I use CPU.

I saw that when loss became nan, memory at GPU was fully allocated


[SSR] HTMLElement is not defined

Describe the bug
Does this library support SSR ?
I got the error
HTMLElement is not defined

when using SSR.

that's a bug @revskill10 i will fix that very soon. thanks for reporting

can you please help test out this version?
npm i<EMAIL_ADDRESS>

hey @revskill10, i have published the fix https://github.com/bluebill1049/react-hook-form/releases/tag/v2.1.15. going to close this issue. feel free to reopen if found related issue.


updated install-docs/readme.md further-steps

don't approve until #688 is completed
added new commands + aliases
removed --reply
changed $(source:name) to $(source:username_raw)
added $(tb:broadcaster)
added $(tb:bot_domain)
Pull request checklist:

[X] CHANGELOG.md was updated, if applicable

$(source:name) is equivalent to $(source:username_raw) - they are both the display name.
Why remove --reply? Reply means if the command is whispered e.g. by mods/the bot owner the response will be a whisper as well instead of a "say" to chat.

Apart from that, the other points are good improvements that I'll happily accept! 🚀

$(source:name) is equivalent to $(source:username_raw) - they are both the display name.
Why remove --reply? Reply means if the command is whispered e.g. by mods/the bot owner the response will be a whisper as well instead of a "say" to chat.

I was mistaken on the part of --reply which is why I did already re-include it. ebfe1a5
As for $(source:name) vs $(source:username_raw) I was under the impression that $(source:username_raw) took capitalization into consideration and $(source:name) did not, which I have realized is not true so I will revert. 👍

despite what github thinks the changelog is correct
probably the 🤠 way I did it though

username_raw has become name, and username has become login, because that‘s what twitch calls it these days. https://github.com/pajbot/pajbot/blob/1c500fd88bbbd9c3f22986ba1df16a0af9af294c/docs/variables.txt#L115


Upgrade toran proxy to version 1.4.4

Bump, see https://github.com/cedvan/docker-toran-proxy/issues/35

@cedvan I'd be willing to take over updating the repo on a regular schedule, if you could give me merge access. Right now we should make sure that updates get merged on a regular base.

@cedvan can @jeremyzahner take over? Unfortunately we're going to have to start to look for alternative images if this doesn't stay updated :(

Hi @adamlc,
I am in holiday tomorrow, I will update this image :wink:

Pull request is outdated : https://github.com/cedvan/docker-toran-proxy/releases/tag/1.4.4

Thank you for contributing


HADOOP-18918. ITestS3GuardTool fails if SSE/DSSE encryption is used

Endpoint: us-west-2
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch (configured with encryptions: SSE, DSSE and NONE)

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 35s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
41m 21s

trunk passed

+1 :green_heart:
compile
0m 45s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
0m 38s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 36s

trunk passed

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 32s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 38s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 12s

trunk passed

+1 :green_heart:
shadedclient
33m 11s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 29s

the patch passed

+1 :green_heart:
compile
0m 34s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
0m 34s

the patch passed

+1 :green_heart:
compile
0m 27s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
0m 27s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 22s

the patch passed

+1 :green_heart:
mvnsite
0m 33s

the patch passed

+1 :green_heart:
javadoc
0m 17s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m  5s

the patch passed

+1 :green_heart:
shadedclient
33m 41s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m 48s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

125m 59s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6165

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux b413114cafde 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 766c4229bc7218234bbffe47cc7e52089f20f1dd

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/testReport/

Max. process+thread count
560 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ahmarsuhail @mehakmeet @mukund-thakur could you please review this PR?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 31s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m  7s

trunk passed

+1 :green_heart:
compile
0m 41s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 34s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 31s

trunk passed

+1 :green_heart:
mvnsite
0m 41s

trunk passed

+1 :green_heart:
javadoc
0m 27s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 34s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m  8s

trunk passed

+1 :green_heart:
shadedclient
32m  1s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 29s

the patch passed

+1 :green_heart:
compile
0m 33s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
0m 33s

the patch passed

+1 :green_heart:
compile
0m 24s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
0m 24s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 19s

the patch passed

+1 :green_heart:
mvnsite
0m 31s

the patch passed

+1 :green_heart:
javadoc
0m 15s

the patch passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m  6s

the patch passed

+1 :green_heart:
shadedclient
32m 17s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
2m 44s
/patch-unit-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 37s

The patch does not generate ASF License warnings.

122m 45s

Reason
Tests

Failed junit tests
hadoop.fs.s3a.impl.TestErrorTranslation

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6165

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 4e8c05fd5174 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 1d3be912abc36a34a8bf3cf891fa09dd54f02ec3

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/testReport/

Max. process+thread count
558 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

TestErrorTranslation seems recent regression, the base branch of this PR does not even have the test.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 32s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
46m 12s

trunk passed

+1 :green_heart:
compile
0m 42s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 33s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 30s

trunk passed

+1 :green_heart:
mvnsite
0m 41s

trunk passed

+1 :green_heart:
javadoc
0m 24s

trunk passed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 33s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 12s

trunk passed

-1 :x:
shadedclient
49m 14s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 22s
/patch-mvninstall-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

-1 :x:
compile
0m 22s
/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-aws in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
javac
0m 22s
/patch-compile-hadoop-tools_hadoop-aws-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-aws in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
0m 23s
/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
javac
0m 23s
/patch-compile-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 55 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-1 :x:
blanks
0m  0s
/blanks-tabs.txt
The patch 624 line(s) with tabs.

-0 :warning:
checkstyle
0m 20s
/buildtool-patch-checkstyle-hadoop-tools_hadoop-aws.txt
The patch fails to run checkstyle in hadoop-aws

-1 :x:
mvnsite
0m 23s
/patch-mvnsite-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

-1 :x:
javadoc
0m 22s
/patch-javadoc-hadoop-tools_hadoop-aws-jdkUbuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.txt
hadoop-aws in the patch failed with JDK Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04.

-1 :x:
javadoc
0m 23s
/patch-javadoc-hadoop-tools_hadoop-aws-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt
hadoop-aws in the patch failed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05.

-1 :x:
spotbugs
0m 22s
/patch-spotbugs-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+1 :green_heart:
shadedclient
4m 29s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 23s
/patch-unit-hadoop-tools_hadoop-aws.txt
hadoop-aws in the patch failed.

+0 :ok:
asflicense
0m 22s

ASF License check generated no output?

109m  8s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6165

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 1266bcef325d 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / b0e8436c25fe5444066d3a0d7d4a5bc670153865

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-<IP_ADDRESS>+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/testReport/

Max. process+thread count
160 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6165/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


A handful of fixes and refinements - Devel is gone and iris crashes should be done for.

Chop chop.

Looks mostly fine for me. Since the Zonked Mixins are gone the StatusEffects class can be cleaned up, too - if you are not planning replacing it with an other effect, that is.

Next release should have major bumped

I am not sure what exactly there is to backport. Hmm, I guess yeeting devel stuff?


Workflow for adding and removing primitives

A common workflow in Featuretools is to add and remove primitives to find an optimal set. See this StackOverflow answer for more details.
As a user iterates through primitive options, they have to rerun ft.dfs with the different inputs. This means the feature calculations are often repeated. To improve the time it takes to do this workflow, I propose that a user optionally be able to provide a feature matrix from a previous run of DFS. If a feature for a new run already exists from a prior run, it wont be recalculated.
For example
# Normal DFS call
feature_matrix, feature_defs = ft.dfs(entityset=es,
                                      target_entity="customers",
                                      agg_primitives=["count"],
                                      trans_primitives=["month"])

Then a user adds a new primitive. This should only calculate the new features
feature_matrix, feature_defs = ft.dfs(entityset=es,
                                      target_entity="customers",
                                      agg_primitives=["count", "sum"],
                                      trans_primitives=["month"],
                                      feature_matrix=feature_matrix)

Finally, the user decides that the month primitive is not needed. This change will not trigger any recalculation because we are only removing features
feature_matrix, feature_defs = ft.dfs(entityset=es,
                                      target_entity="customers",
                                      agg_primitives=["count", "sum"],
                                      trans_primitives=[],
                                      feature_matrix=feature_matrix)

If a user removed the feature_matrix argument from any of the above calls the return values would not change.

Is this working right now? Or it will be developed?
It will be very usefull.

@dlionbox it isn't implemented at the moment as we're not sure how much people want it. if more people chime in that they want it, we'd be happy to build it!

@kmax12 thank you for answering! I'll be happy using it


BUG: statsmodels_to_results suffers from inaccuracy

Using .summary() is nice for printing but bad for numerical computing, so we probably shouldn't use it to get actual values in statsmodels_to_results. By way of example:
>>> ch_model = smf.mixedlm(
...     "theta ~ -1 + ch_name:Condition",
...     ch_summary, groups=ch_summary["subject"]).fit(method='powell')
>>> ch_model_df = mne_nirs.statistics.statsmodels_to_results(ch_model)
>>> print(np.array(ch_model_df['P>|z|'])[17:33])
[0.    0.135 0.541 0.89  0.146 0.033 0.221 0.19  0.191 0.002 0.112 0.018
 0.129 0.    0.952 0.   ]
>>> print(np.array(ch_model.pvalues)[17:33])
[2.07866441e-07 1.35295118e-01 5.40928010e-01 8.90354390e-01
 1.46297283e-01 3.25360602e-02 2.20535361e-01 1.90387112e-01
 1.91051924e-01 2.39444468e-03 1.11936842e-01 1.77914955e-02
 1.29375584e-01 1.17267783e-04 9.52492330e-01 5.13254033e-05]

I think in the current code, any p-value less than 0.0005 significance will becomes 0.0 due to imprecision.

Great catch. I can look at this next week. Or do you want to submit a fix?
I'm loving that you're using this and finding issues. This api is version 1, but I have plans/ideas for improvements to work on this year. So please feel free to raise issues for any small frustrations too.

But as you've said to me before. First make it correct, so I want to squish as many accuracy bugs as possible and get robust tests before changing api.

Or do you want to submit a fix?

There are several numerical values, probably all of them need to be replaced. Not sure what the best way to do this is -- currently the .summary() table is used, I guess a mapping between the columns of that table that need to be replaced and their replacement attributes like {'P>|z|': 'pvalues', 'P>|t|': 'pvalues', ...} would work. I can make a PR for this next week but feel free to tackle it before then (this way or some other way) if you want


Remove building the contract from the migration cheat codes

Reasons:

providing compiler configuration (e.g. currently cairo-paths are not respected)
preserving compilation output
unix philosophy violations

wasn't this fixed some time ago?


Support for reading generic python file objects?

Hi,
I've got a database of about 30Gb of gzip-compressed RAW images that I need to process. I'd like to avoid decompressing the files on disk, and instead just open each file as a file object with python's gzip module and then feed that to rawpy. However, RawPy will only accept a file path, which it then handles opening internally.
From skimming the code I couldn't tell whether or not this was a fundamental limitation of the underlying libraw. It also occurred to me that if libraw operates at a low-level on file pointers, the python-land object file from the gzip module might have no way to communicate w/ the c(++) layer.
Could the author comment on whether this functionality would be feasible to implement? If it's not simple, I will probably just content myself with the somewhat hackish workaround of unzipping each file to a temp location in order to read it with rawpy, but it's not ideal.

It would indeed be a good idea if rawpy could handle arbitrary File-like objects. Note that gzip does not give you an actual file pointer but just a File-like object since all the unpacking happens in memory and not on disk with real operating system File handles. Supporting arbitrary File-like objects would be possible by using libraw's LibRaw_buffer_datastream. This would mean reading all data from the Python File-like object into a buffer, and then wrapping it in a LibRaw_buffer_datastream object so that the raw file can be opened with LibRaw::open_datastream instead of LibRaw::open_file.
I'm not completely sure how hard it will be to implement properly. The difficulty is always making sure that the C-level stuff is managed correctly (references, garbage collection, ...) so that there are no memory leaks or seg faults. If there is someone else who needs that feature as well then I'm happy to have a look at it, just a bit busy currently.

I implemented this functionality now, wasn't that hard in the end. It will be part of the next release.

A new version has been released which includes this feature.


Cached subgeneric names rendering is not capturing italics correctly?

@proceps I think this is likely fixed, we just need to re-index the data?

I checked, it is a real one. Probably related only to generic combination, and since we could not create those before, it was never noticed.


How to debug flyway maven plugin on huge dataset?

Hi,
I'm experiencing a weird behaviour of flyway when working with huge dataset.
The dataset file contains more then > 100000 entries. They are commits for each 1000 entries.
I'm running the migration against the remote Oracle 11.1 through maven plugin.
There is no any error, the data is inserted, but maven stacks for ever. There is no activity (CPU/memory) accordingly to VisualVM attached to this process.
How can I debug this situation?

You can debug Maven (and its plugins) as described here: http://tryitnw.blogspot.de/2013/04/tips-remote-debug-with-maven-3-plug-in.html
It may be easier to run the same dataset using flyway-core and a tiny java app and debug that.


Added an orbit cam using Dolly to the Scene Viewer

Checklist

CI Checked:

[x ] cargo fmt has been ran
[ ] cargo clippy reports no issues

  Checking bimap v0.6.3
error: incorrect implementation of `clone` on a `Copy` type
  --> rend3-types/src/lib.rs:59:29
   |
59 |       fn clone(&self) -> Self {
   |  _____________________________^
60 | |         Self {
61 | |             idx: self.idx,
62 | |             _phantom: PhantomData,
63 | |         }
64 | |     }
   | |_____^ help: change this to: `{ *self }`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#incorrect_clone_impl_on_copy_type
   = note: `#[deny(clippy::incorrect_clone_impl_on_copy_type)]` on by default

[x] cargo test succeeds
[x] cargo rend3-doc has no warnings
[ ] cargo deny check issues have been fixed or added to deny.toml

warning[unmatched-skip-root]: skip tree root was not found in the dependency graph
   ┌─ /Users/andy/rust/rend3d/rend3/deny.toml:29:5
   │
29 │     { name = "loom", version = "0.5" },
   │     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no crate matched these criteria

warning[license-not-encountered]: license was not encountered
  ┌─ /Users/andy/rust/rend3d/rend3/deny.toml:4:42
  │
4 │ allow = ["Apache-2.0", "MIT", "MPL-2.0", "LicenseRef-UFL-1.0", "Unlicense"]
  │                                          ^^^^^^^^^^^^^^^^^^^^ unmatched license allowance

Manually Checked:

[ ] relevant examples/test cases run
[ ] changes added to changelog

[ ] Add credit to yourself for each change: Added new functionality @githubname.
[ ]

Suggestion:
Added an orbit cam using the Dolly crate to the Scene Viewer @apps4uco

Related Issues
#307
Description
Added an orbit cam using Dolly to the Scene Viewer @apps4uco.
Keys:
W Forward
S  Back
A Left
D Right
Q Up
Z Down
Note: Mouse movement is supported but a bit jerky
Tested with default sample and https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0/ABeautifulGame

This is awesome! Might take me a minute to review this, but I've wanted to integrate dolly for a while!

By the way I am happy to work on this functionality more e.g.: to implement other types of cameras
if you would like.

Sorry for the massive delay in getting back to this.
This is a good first start! It's actually not a bad way of navigating. My thoughts so far:

It would be nice if wasdqz still moved the center of the camera, and mouse + mousewheel did orbit size. I think that would be a more natural way to explore spaces
The sensitivity of the mouse seems way high


Add data preparation utility for hierarchical time series

Issue #, if available:
Description of changes:
This PR implements the missing functionality for handling hierarchical/grouped time series data and training hierarchical models correctly without data-preprocessing issues (see #1967 and #2067).  It also provides a tutorial on loading hierarchical time series and training and evaluation of the hierarchical model DeepVARHierarchical.
Details:

The user has to provide only the bottom-level time series (pandas dataframe) and the aggregation matrix (numpy ndarray), which are the minimal requirements in constructing hierarchical or even grouped time series.  The aggregated time series are then internally constructed and all the time series in the hierarchy are available as a pandas dataframe.

The hierarchical time series thus constructed can be easily converted to a Dataset on which mini-batch training can be done, thanks to the newly implemented PandasDataset.

The tutorial explains the procedure for creating hierarchical time series, training the model and doing evaluation via backtesting by creating the train-test split explicitly (i.e., not using make_evaluation_predictions or backtest_metrics).

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

@rshyamsundar you also need to add the tutorial to the table of contents in the index.md in the same directory as the .md.template file

I think this PR is in great shape. One thing that could make it even better is to include an example (maybe like a 3rd section after Model Training and Forecasting, and Model Evaluation via Backtesting), where dynamic features are included. I suggest this as it is one element from gluonts.dataset.hierarchical.to_pandas_dataset that remains a tiny bit unclear how to use, and one can expect questions about this in the future.

I think this PR is in great shape. One thing that could make it even better is to include an example (maybe like a 3rd section after Model Training and Forecasting, and Model Evaluation via Backtesting), where feat_dynamic_real are included. I suggest this as it is one element from gluonts.dataset.hierarchical.to_pandas_dataset that remains a tiny bit unclear how to use, and one can expect questions about this in the future.

Hi @melopeo Here is the PR that addresses your comments. Thanks!


basic drain and into_iter implementations

I just took a quick look at implementing drain and into_iter.
The tests seem to pass, but that might just be luck.
This code should probably not be merged, since it seems horrendously inefficient, but maybe it can be used to work towards an actual implementation.

I'm going to close this for the time being, and then we can re-open if you find time to pick it up again @soruh :)

@jonhoo Great, thank you. I'll probably still be quite busy for a while but I should be able to contribute to discussions regarding this.


Amplify build is ignoring my @key and @connection queries

Describe the bug
On my schema.graphql I added models with @key and @connection directives, after that I followed the instructions from the docs to update my api: amplify api gql-compile and then amplify push. Everything works fine, I have all the queries I need and I can call/use them correctly.
But when I go to the AWS Console and connect my branch and my backend, the CD runs and all my queries generated by @key and @connection are erased, then if try to call then  I got an error telling me they are invalid:
"Validation error of type FieldUndefined: Field 'querySessionsByCreatedAt' in type 'Query' is undefined @ 'querySessionsByCreatedAt'"
I don't know if my schema has something wrong and why only works using amplify push but here are all my files:
schema.graphql, I have a @key to sort Sessions by createdAt and @connection to handle a many to many relation between Sessions and SessionsGroups.
type Todo @model
  @auth(rules: [
    { allow: owner },
    { allow: groups, groups: ["admin"], operations: [read] }
])
{
  id: ID!
  title: String!
  status: String
  createdAt: String!
  type: String!
}

type Session @model
  @key(name: "byCreatedAt", fields: ["type", "createdAt"])
{
  id: ID!
  name: String!
  link: String!
  type: String!
  createdAt: String!
  sessionGroups: [SessionSessionGroup] @connection(keyName: "bySession", fields: ["id"])
}

type SessionGroup @model @auth(rules: [
  { allow: groups, groups: ["admin"] },
  { allow: groups, groupsField: "userGroups", operations: [read, update] }
])
 {
  id: ID!
  name: String!
  userGroups: [String]
  sessions: [SessionSessionGroup] @connection(keyName: "bySessionGroup", fields: ["id"])
}

type SessionSessionGroup
  @model(queries: null)
  @key(name: "bySession", fields: ["sessionID", "sessionGroupID"])
  @key(name: "bySessionGroup", fields: ["sessionGroupID", "sessionID"])
{
  id: ID!
  sessionID: ID!
  sessionGroupID: ID!
  session: Session! @connection(fields: ["sessionID"])
  sessionGroup: SessionGroup! @connection(fields: ["sessionGroupID"])
}

Here is my amplify.yml
version: 0.1
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - npm install
        - npm run test
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: dist
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
test:
  phases:
    preTest:
      commands:
        - npm ci
        - npm install
        - npm install wait-on
        - 'npm start & npx wait-on http://localhost:8080'
    test:
      commands:
        - 'npx cypress run'
  artifacts:
    baseDirectory: cypress
    configFilePath: 'cypress.json'
    files:
      - '**/*.csv'
      - '**/*.mp3'

Amplify CLI Version
4.21.2
To Reproduce
Create a project, add the user auth (default options), then I add API with the schema above, run amplify push, then connect the branch and the new backend
Expected behavior
Expected the deploy from CD has the same behavior from amplify push, so all the necessary queries should be there.
Desktop (please complete the following information):

Mac OS Catalina 10.15.3
Node: v12.16.1
npm 6.14.5

Additional context
Things that I already tried:

Delete my API and added it again (didn't work)
After all that I run amplify codegen, it looks like it pulls some info from the Api on the cloud, and erase my queries locally as well. Queries that are erased: querySessionSessionGroupsBySessionGroup, querySessionSessionGroupsBySession and querySessionsByCreatedAt

Steps when I tried the create the API again:
❯ amplify add api
? Please select from one of the below mentioned services: GraphQL
? Provide API name: apiname
? Choose the default authorization type for the API Amazon Cognito User Pool
Use a Cognito user pool configured as a part of this project.
? Do you want to configure advanced settings for the GraphQL API No, I am done.
? Do you have an annotated GraphQL schema? No
? Do you want a guided schema creation? Yes
? What best describes your project: Single object with fields (e.g., “Todo” with ID, name, description)
? Do you want to edit the schema now? Yes
Please edit the file in your editor: /path/to/my/projectamplify/backend/api/apiname/schema.graphql
? Press enter to continue
? Do you want to generate code for your newly created GraphQL API Yes
? Choose the code generation language target javascript
? Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
? Do you want to generate/update all possible GraphQL operations - queries, mutations and subscriptions Yes
? Enter maximum statement depth [increase from default if your schema is deeply nested] 5

I realized that on my build the last version is installed, 4.22, then I updated my locally version and I got the same results locally, my queries were not generated. Is this a bug? Is there any config missing?
I received all these warning while the last versions was being installed:
$ npm install -g @aws-amplify/cli@latest

npm WARN deprecated<EMAIL_ADDRESS>Merge GraphQL Schemas has been deprecated and merged into GraphQL Tools, so it will no longer get updates. Use GraphQL Tools instead to stay up-to-date! Check out https://www.graphql-tools.com/docs/migration-from-merge-graphql-schemas for migration and https://the-guild.dev/blog/graphql-tools-v6 for new changes.
npm WARN deprecated<EMAIL_ADDRESS>GraphQL Toolkit is deprecated and merged into GraphQL Tools, so it will no longer get updates. Use GraphQL Tools instead to stay up-to-date! Check out https://www.graphql-tools.com/docs/migration-from-toolkit for migration and https://the-guild.dev/blog/graphql-tools-v6 for new changes.
npm WARN deprecated<EMAIL_ADDRESS>GraphQL Toolkit is deprecated and merged into GraphQL Tools, so it will no longer get updates. Use GraphQL Tools instead to stay up-to-date! Check out https://www.graphql-tools.com/docs/migration-from-toolkit for migration and https://the-guild.dev/blog/graphql-tools-v6 for new changes.
npm WARN deprecated<EMAIL_ADDRESS>GraphQL Toolkit is deprecated and merged into GraphQL Tools, so it will no longer get updates. Use GraphQL Tools instead to stay up-to-date! Check out https://www.graphql-tools.com/docs/migration-from-toolkit for migration and https://the-guild.dev/blog/graphql-tools-v6 for new changes.
npm WARN deprecated<EMAIL_ADDRESS>https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated<EMAIL_ADDRESS>core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated<EMAIL_ADDRESS>GraphQL Import has been deprecated and merged into GraphQL Tools, so it will no longer get updates. Use GraphQL Tools instead to stay up-to-date! Check out https://www.graphql-tools.com/docs/migration-from-import for migration and https://the-guild.dev/blog/graphql-tools-v6 for new changes.
npm WARN deprecated<EMAIL_ADDRESS>request has been deprecated, see https://github.com/request/request/issues/3142
/usr/local/bin/amplify -> /usr/local/lib/node_modules/@aws-amplify/cli/bin/amplify

Hi @LucasAndrad. For the querySessionSessionGroupsBySessionGroup and querySessionSessionGroupsBySession queries, I would expect them to not be generated since the SessionSessionGroup model has queries turned off with @model(queries: null). I'm not sure yet why querySessionsByCreatedAt isn't being generated though.

But querySessionSessionGroupsBySessionGroup and querySessionSessionGroupsBySession should be generated because of the @connection (the docs said that here) and they are generated correctly when I use version 4.21.2, but when I update to version 4.22 they are not generated anymore.

Looking at the example in the docs, the queries listed are not on the PostEditor type (which sets queries to null), rather they query the User or Post directly.
It's possible I'm misunderstanding something here, but if you set the queries to null on the @model then I think those queries should not have been generated in the previous version either.

I tested my self @model(queries: null) only blocks the generation of getItemName and listItemName

I can confirm the behavior @LucasAndrad is describing here. I also noticed that my auth directives like @aws_iam @aws_cognito_user_pools were not being applied to my generated schema until I removed queries: null from the @model.
That being said, I'm not quite certain queries: null makes a whole lot of sense as a recommended approach. Same for mutation/subscription: null. Mainly because although its a generation command, it's used in practice as a security mechanism.

Thanks for sharing your case @r0zar but as you can see, we need to wait a lot of months before get any feedback from amplify team. When I opened this issue this repository already had 620 issues (a lot) now they have 764.
Here is what I did to get my queries to work, I added this backend pre-build step on my amplify.yml to force the build to use version 4.21, the bad thing is that I can't update amplify anymore, so I hope they fix this bug before major changes in future versions.
backend:
  phases:
    preBuild:
      commands:
        - npm install -g<EMAIL_ADDRESS>

@LucasAndrad @r0zar
I reproduced the issues you posted here, those extra queries were indeed generated before version 4.22, the @key transformer used to auto generate query operations, we have since disabled it.
If you still want to generate the queries, you can specify the "queryField" in the @key definition.
https://docs.amplify.aws/cli/graphql-transformer/directives#key

Okay, good to know! I'm going to test queryField in a key and queries: null in the model to make sure it works as expected. If that checks out, this issues is resolved for me.

Closing the issue for now, please open a new issue if you are still blocked.


Prevent panic during wrappedConn close at hammertime

When reviewing #10549 there is a subtle race during forced shutdown/hammertime that can lead to a panic. This is relatively benign - but it's clearly distracting from the real problem.
Please note this is not the cause of #10549 and will not fix that issue - that issue is a deadlock within the DB.
Signed-off-by: Andrew Thornton<EMAIL_ADDRESS>

Lint error

Codecov Report

Merging #11219 into master will increase coverage by 0.00%.
The diff coverage is 12.50%.

@@           Coverage Diff           @@
##           master   #11219   +/-   ##
=======================================
  Coverage   43.19%   43.20%
=======================================
  Files         604      604
  Lines       86050    86056    +6
=======================================
+ Hits        37170    37180   +10
+ Misses      44288    44286    -2
+ Partials     4592     4590    -2

Impacted Files
Coverage Δ

modules/markup/markdown/goldmark.go
37.10% <ø> (+0.14%)
:arrow_up:

modules/graceful/server.go
46.93% <12.50%> (-2.35%)
:arrow_down:

modules/indexer/stats/queue.go
62.50% <0.00%> (-18.75%)
:arrow_down:

modules/indexer/stats/db.go
40.62% <0.00%> (-9.38%)
:arrow_down:

modules/git/utils.go
65.67% <0.00%> (-4.48%)
:arrow_down:

modules/git/repo.go
51.04% <0.00%> (+0.41%)
:arrow_up:

models/issue.go
51.86% <0.00%> (+0.46%)
:arrow_up:

modules/notification/webhook/webhook.go
40.20% <0.00%> (+0.58%)
:arrow_up:

modules/queue/unique_queue_disk_channel.go
55.76% <0.00%> (+1.92%)
:arrow_up:

services/pull/patch.go
69.23% <0.00%> (+3.49%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f959ac...50a4b25. Read the comment docs.


A brief getting started, an example would be welcomed!

And what is "standard ar tag"?
Thanks.

Hey eighteight,
Thanks for the interest/question. You're right in being confused regarding the term, as no such real thing exists. What I'm referring to is the most often used thick border with binary pattern contained inside, somewhat ala a QR code, except with unambiguous binary patterns being used as opposed to the three identifying corners.
Your advice in the headline is duly noted. The project would ideally have both a 'getting started' and an example. As you can tell by the commit frequency, the project is on a current hiatus, but I do plan on getting back and at least completing it in a base-form.


Add automatic instructions

This pull request adds instructions for how to use a git pre-commit hook to automatically sort the package.json file.

Nice! Thanks!

:tada: This PR is included in version 1.23.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


chore(package): update npm-template-sync to version 7.9.5

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


[Feat] First shot at ARNEIS System Architecture document

Closes #94
📑 Description

✅ Checks

[x] My pull request adheres to the code style of this project
[ ] My code requires changes to the documentation
[x] I have updated the documentation as required
[x] All the tests have passed

ℹ Additional Information

Looks Good (enough) To Me, I feel no harm in merging this PR.
Will improve the document in future PR.


No response in initiating Kick-In, Corner-Kick, Goal-Kick

We were testing our code of bhuman-2023 today, but we found that when we initiate a Kick-In, Corner-Kick or Goal-Kick to a certain team, the robots had no response to it, which meant that they were still at 'playing' state. Nevertheless, when we initiate them in Simrobot, it could successfully turn into the certain mode and complete the progress. Is there any problem with the GameController3 or our configurations?
Thanks for your reply!

Did you compile the GameController from a source code version after March 14? Those versions will not be compatible with the B-Human 2023 code release because the rules changed.
Are you sure that the network configuration is correct? If you didn't change the B-Human network configuration, the GameController computer must have an IP address in the <IP_ADDRESS>/16 network on the interface that is used to communicate with the robots. The robots have the team number 89 which must be configured in config/teams.yaml and config/<league>/teams.yaml first.

Generally, if the robots do not show up as active (which means that the red dot next to the robot numbers do not turn green), there is a problem with communication between the GameController and the robots.


slack notification

Notification support can be added like below if any action taken by kube-pod-terminator:
      --slack-channel string                slack channel for reboot notfications
      --slack-hook-url string               slack hook URL for reboot notfications
      --slack-username string               slack username for reboot notfications

kured can be examined for solution

dummy comment

/remove lifecycle-stale


fix encoding problem and unit test

fix encoding problem in EVENT page when Event carries Chinese data (may happens in windows ISSUE #1040, the NativeMessageCodec doesn't decode Chinese well, when the server receives data via Netty, and the readString method uses new String without Charset indicated, java will use system default Charset, which is not reliable).
fix encoding problem in Handler, the compress method may has the same problem as above. Despite the server decodes message correctly, compress messes up the encoding.
fix encoding problem in HtmlMessageCodec, Netty decode Chinese correctly but it still not safe. now  the data supports Chinese.
fix some unit test. the MessageQueue is apparently not Initializable.
impove Cat.getCatHome, to prevent someone who forgets CAT_HOME ends with "/", otherwise the cat server will try to lookup some file in the path like /some/path/appdatas/catbucket.

note that the URIEncoding="utf-8" is added, the problem is still there.
Here are some pictures of encoding problem described above.
before:

after:

The Chinese word "联通" has a well-know story:
in the windows :  enter the word in Notepad save and reopen it, the encoding is fail. So I use this word.
Claim: there is no other meaning with the company.
The code will keep a principle:
The type and name will not support Chinese.

@stdrickforce Please review the PR, thanks.

谢谢，赞，这个我们double check下，多加关注