Euler angle conventions and Body input/output angles

Hi all,
I installed MoorDynv2 matlab wrapper. I was testing input and output angles for Body object.
lines_test.txt
I am modelling a single Body as coupled object and trying to provide motion externally calculated.
the script I am using is below:
clc;close all;clear
%% Create MoorDyn system
system1 = MoorDynM_Create('lines_test.txt');
%% find num of DOF
DOF =MoorDynM_NCoupledDOF(system1);
x = zeros(1,DOF);
dx = zeros(1,DOF);
%% Setup the initial condition
MoorDynM_Init(system1, x, dx);
%% Simulate
dt =0.04;
t = 0:dt:10;
x =zeros(length(t),DOF);
dx =zeros(length(t),DOF);
% provide roll angle ansd speed
roll_speed = 5; % deg/sec
x(:,4) = t*roll_speed;
dx(:,4) = roll_speed;
% run the moordyn
tic
for i=1:length(t)-1
[sim_time, f] = MoorDynM_Step(system1, x(i,:), dx(i,:), t(i),dt);
end
%% Alright, time to finish!
MoorDynM_Close(system1);
toc
%% plot
data=readtable('lines_test_Body1.out','FileType','text');
VariableNames = {'Surge','Sway','Heave','Roll','Pitch','yaw'};
indx =4;
figure(1);plot(t,x(:,indx))
hold on
plot(t,data{:,1+indx})
hold off
title(VariableNames{indx})
I am only giving 5 deg/s angular speed in roll. Since there is no other element, i was expecting the roll angle will be same for input and ouput. Not only the roll angles saved by MoorDyn is quite different than expected, but unexplained pitch and yaw angles.

Can anyone explain whats going on here ? Thanks in advance.

Firstly I believe that MoorDyn expects angular values to be in radians (angles are radians, angular velocities are rad/s). So you are specifying a velocity of 5 radians per second, which looks like about what the graph is showing.
Secondly, MoorDyn internally uses quaternions to represent rotation, and converts to and from euler angles when taking input or returning output. This lets MoorDyn avoid any issues of gimbal lock. But this does mean that the exact set of angles used to define a rotation are not necessarily preserved between input and output.
It just so happens that with the XYZ euler angle convention that MoorDyn uses, the angles
roll = 180 deg, pitch = 0 deg, yaw = 0 deg
and
roll = 0 deg, pitch = 180 deg, yaw = 180 deg
are in fact the same rotation. And so when the simulation reaches the point where roll = 180, the quaternion to euler angle conversion inside of MoorDyn ends up spitting out the second set of Euler angles instead of the first.
This online calculator I have found helpful in working with 3d rotations
https://www.andre-gaschler.com/rotationconverter/
In your specific case, besides the lack of a degrees to radians conversion, you probably want to fully specify the body orientation and not just the roll component. That should properly specify the motion you want, but it doesn't solve the issue of the orientation reported by MoorDyn potentially being different from the one you specified. In general solving that problem is challenging, but one way to address it would be to do your own conversion of the rotation values reported by MoorDyn into something like an axis-angle representation. Given that you are always rotating around the x axis this would give you the current angle the object is rotated around the x axis.
Hope this helps, and feel free to follow up with any additional questions or concerns if things still look suspicious to you.

@AlexWKinley thanks for this explanation, I had been wondering about this a bit.
@hamzaJayy I would add that while the function inputs are radians, the MoorDyn input files are still in degrees. This is a convention that comes from MoorDyn's coupling with OpenFAST.

Thanks guys.
I changed the input angles to radians. I kept the angular velocity to zero. I provided x-y-z intrinsic euler angles.
Few things:

Yes, the input for MoorDynM_Step is in radians. And the ouput for MoorDynM_GetBodySate is also in radians. But the output file (.out), generated by input file (.txt), stores angles in degrees. That I assume that's FAST convention.
I understand now the gimbel lock and issues with that. But now as you can see in graph below: something strange happens when roll angle becomes negative. Is it also expected issues with euler rotation or is it something else?

Okay so after some research I have tracked down the source of the confusion.
Firstly, so far as I can tell, the euler angles moordyn is returning are valid representations of the angles you are using.
The reason roll doesn't go negative is because that is apparently the behavior of Eigen function that moordyn uses to convert a rotation matrix to euler angles.
If you look at the documentation for the eulerAngle() function here:
https://libeigen.gitlab.io/docs/group__Geometry__Module.html#ga5a61d4a29ac24ace7043a6301c21e52c
it says
The returned angles are in non-canonical ranges [0:pi]x[-pi:pi]x[-pi:pi].
For canonical Tait-Bryan/proper Euler ranges, use canonicalEulerAngles.

It looks like that canonicalEulerAngles function is relatively new to Eigen, being added in May of this year
and so is not available in the version of Eigen included with moordyn.
Long story short, the reason the roll seems constrained to a weird range is because it's constrained to a weird range.
I'm fairly confident that the rotation of the object is still behaving correctly, but I agree it's a strange result to get.
@RyanDavies19 If we wanted to switch to a canonicalEulerAngles approach, we can just grab that specific function from Eigen and repurpose it for our needs.
moordyn::vec3
canonicalEulerAngles(const moordyn::quaternion& quat, int a0, int a1, int a2)
{
	moordyn::mat3 coeff = quat.toRotationMatrix();
	moordyn::vec3 res{};
	using Index = int;
	using Scalar = moordyn::real;
	const Index odd = ((a0 + 1) % 3 == a1) ? 0 : 1;
	const Index i = a0;
	const Index j = (a0 + 1 + odd) % 3;
	const Index k = (a0 + 2 - odd) % 3;
	if (a0 == a2) {
		// Proper Euler angles (same first and last axis).
		// The i, j, k indices enable addressing the input matrix as the XYX
		// archetype matrix (see Graphics Gems IV), where e.g. coeff(k, i) means
		// third column, first row in the XYX archetype matrix:
		//  c2      s2s1              s2c1
		//  s2s3   -c2s1s3 + c1c3    -c2c1s3 - s1c3
		// -s2c3    c2s1c3 + c1s3     c2c1c3 - s1s3
		// Note: s2 is always positive.
		Scalar s2 = Eigen::numext::hypot(coeff(j, i), coeff(k, i));
		if (odd) {
			res[0] = atan2(coeff(j, i), coeff(k, i));
			// s2 is always positive, so res[1] will be within the canonical [0,
			// pi] range
			res[1] = atan2(s2, coeff(i, i));
		} else {
			// In the !odd case, signs of all three angles are flipped at the
			// very end. To keep the solution within the canonical range, we
			// flip the solution and make res[1] always negative here (since s2
			// is always positive, -atan2(s2, c2) will always be negative). The
			// final flip at the end due to !odd will thus make res[1] positive
			// and canonical. NB: in the general case, there are two correct
			// solutions, but only one is canonical. For proper Euler angles,
			// flipping from one solution to the other involves flipping the
			// sign of the second angle res[1] and adding/subtracting pi to the
			// first and third angles. The addition/subtraction of pi to the
			// first angle res[0] is handled here by flipping the signs of
			// arguments to atan2, while the calculation of the third angle does
			// not need special adjustment since it uses the adjusted res[0] as
			// the input and produces a correct result.
			res[0] = atan2(-coeff(j, i), -coeff(k, i));
			res[1] = -atan2(s2, coeff(i, i));
		}
		// With a=(0,1,0), we have i=0; j=1; k=2, and after computing the first
		// two angles, we can compute their respective rotation, and apply its
		// inverse to M. Since the result must be a rotation around x, we have:
		//
		//  c2  s1.s2 c1.s2                   1  0   0
		//  0   c1    -s1       *    M    =   0  c3  s3
		//  -s2 s1.c2 c1.c2                   0 -s3  c3
		//
		//  Thus:  m11.c1 - m21.s1 = c3  &   m12.c1 - m22.s1 = s3
		Scalar s1 = sin(res[0]);
		Scalar c1 = cos(res[0]);
		res[2] = atan2(c1 * coeff(j, k) - s1 * coeff(k, k),
		               c1 * coeff(j, j) - s1 * coeff(k, j));
	} else {
		// Tait-Bryan angles (all three axes are different; typically used for
		// yaw-pitch-roll calculations). The i, j, k indices enable addressing
		// the input matrix as the XYZ archetype matrix (see Graphics Gems IV),
		// where e.g. coeff(k, i) means third column, first row in the XYZ
		// archetype matrix:
		//  c2c3    s2s1c3 - c1s3     s2c1c3 + s1s3
		//  c2s3    s2s1s3 + c1c3     s2c1s3 - s1c3
		// -s2      c2s1              c2c1
		res[0] = atan2(coeff(j, k), coeff(k, k));
		Scalar c2 = Eigen::numext::hypot(coeff(i, i), coeff(i, j));
		// c2 is always positive, so the following atan2 will always return a
		// result in the correct canonical middle angle range [-pi/2, pi/2]
		res[1] = atan2(-coeff(i, k), c2);
		Scalar s1 = sin(res[0]);
		Scalar c1 = cos(res[0]);
		res[2] = atan2(s1 * coeff(k, i) - c1 * coeff(j, i),
		               c1 * coeff(j, j) - s1 * coeff(k, j));
	}
	if (!odd) {
		res = -res;
	}
	return res;
}

And then change Quat2Euler to be
vec3
Quat2Euler(const quaternion& q)
{
	return canonicalEulerAngles(q.toRotationMatrix(), 0, 1, 2);
}

And I believe that would alleviate some of the confusion.

@AlexWKinley I will try that out and see how it performs in some of my test cases. Thanks for the suggestion.

This is resolved by PR #167


frappe-ui not installed after bench add-frappe-ui

Hi @NagariaHussain, I have tried to install frappe-ui in Vue through Doppio.
frappe@WSJSANCHEZ:~/testfrappeui/apps$ bench add-frappe-ui
Dashboard Name [frontend]: frontend
App Name: my_custom_app
Adding Frappe UI starter to my_custom_app...
! destination directory is not empty, aborting. Use --force to override
yarn install v1.22.19
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.16s.
🖥  You can start the dev server by running 'yarn dev' in apps/my_custom_app/frontend
📄  Docs: https://frappeui.com
yarn list frappe-ui:

If you do things wrong, you'll get bad results... my mistake. I'm closing this.


Cleanup ManageIQ domain match instance names to filenames

Purpose or Intent
The Automate export code changes the dots in filenames to underscores. The instance name and the file name are supposed to be consistent. We were converting all dots to underscores. We should have been only converting the dot to underscore only if the instance name starts with dot. e.g. .missing would be converted to _missing when writing to the file system
Links
https://bugzilla.redhat.com/show_bug.cgi?id=1287891
Steps for Testing/QA
QA has a test script that they use to validate that the instance name and the filenames are consistent.

@gmcculloug @tinaafitz
Please Review

@mkanoor
Looks good.

@mkanoor To clarify, if the instance name begins with dot (.) we still convert it to an underscore (_) so the file is not hidden.  All other dot-characters in an instance name will remain unchanged.  Please confirm.
For example:
The .missing method will be _missing in the export,
However, test.instance which use to export as test_instance will now export to test.instance

@gmcculloug
Yes thats correct

@mkanoor @gmcculloug Merge conflict when running git cherry-pick -x -m 1 16aae50
Please create separate PR against Darga branch.

@chessbyte
PR https://github.com/ManageIQ/manageiq/pull/9640 created for Darga branch

Backported via #9640


Binary data

Hi!
I´m facing issues while intercepting/sending body with binary data, even when the interceptor does nothing with the content.
Example:
intercept: (body, send) => {
send(body);
}
I try this with some pdf files and they were all corrupted after the interception. Is there something that I may not paying attention regarding binary content?
Best regards,
Vinicius

Yes, please see this fork https://github.com/fuminchao/express-interceptor


Remove unwanted test

Remove the unwanted version checking test. Tests run fine with Ruby 1.5.2 and 1.6.4 perfectly. So the test is unnecessary.

This build failed, it was here for a good reason :)

But that doesn't make any sense. The commit just removed a test. Altered nothing in the library code. Then how can it cause build failure? AFAIK that test was an isolated one. It doesn't affect or influence the expect statement.
I am confused.

The change removes the pending invocation, which would stop running the example and return it as "pending" (which doesn't count as a failure, hence why the build previously wasn't broken). Since you removed that pending call, the test continues to run and, because Rack was at 1.4.2, fails.
That said, the check seems a little odd (it's okay for this not to work, sometimes?). I think it would be more reasonable to require Rack 1.5.3+ explicitly. However, it seems a bit test-specific, so maybe it really is optional... in which case, the test as-is is probably best.

OK. Since the test works fine for Rack>=1.5.2 (from my experience), I am going to enforce Rack>=1.5.2 requirement in Debian and remove this test from the Debian package. So it will pass, for sure.
About the upstream code, I think we can do version >= 1.5. That is, major>=1 and minor>=5. And remove the release part. Won't that work?

Lets backup. I am ok if Grape requires Rack 1.5 or newer explicitly. And then we should adjust the test as such or remove it.

I am still interested in this provided it's squashed, builds pass, etcetera.

Bump @balasankarc, want to finish this?

Closing this, please feel free to reopen if you want to finish it.


Using flow, calling concat on a Seq with a Map([[V, V]]) causes a type error (v4)

What happened
Not sure if this is a type bug or a deprecated API thing or something else entirely, and I don’t fully understand the original intent of this code, but here’s what I’m seeing:
  var startKey = selectionState.getStartKey();
  var endKey = selectionState.getEndKey();
  var blockMap = contentState.getBlockMap();
  var newBlocks = blockMap
    .toSeq()
    .skipUntil((_, k) => k === startKey)
    .takeUntil((_, k) => k === endKey)
    .concat(Map([[endKey, blockMap.get(endKey)]]))
    .map(operation);

In that code block, where startKey and endKey are strings and blockMap is an OrderedMap of (subclassed) Records, flow gives this error:
Error: src/model/transaction/modifyBlockForContentState.js:32
                       v-------
 32:   var newBlocks = blockMap
 33:     .toSeq()
 34:     .skipUntil((_, k) => k === startKey)
 35:     .takeUntil((_, k) => k === endKey)
 36:     .concat(Map([[endKey, blockMap.get(endKey)]]))
         ---------------------------------------------^ rest parameter array of call of method `concat`. Has some incompatible type argument with
481:   concat<KC, VC>(...iters: Array<Iterable<[KC, VC]> | PlainObjInput<KC, VC>>): KeyedSeq<K | KC, V | VC>;
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ array type. See: node_modules/immutable/dist/immutable.js.flow:481
  Type argument `T` is incompatible:
     36:     .concat(Map([[endKey, blockMap.get(endKey)]]))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Map. This type is incompatible with
    481:   concat<KC, VC>(...iters: Array<Iterable<[KC, VC]> | PlainObjInput<KC, VC>>): KeyedSeq<K | KC, V | VC>;
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ union: type application of identifier `Iterable` | type application of polymorphic type: PlainObjInput. See: node_modules/immutable/dist/immutable.js.flow:481
      Member 1:
      481:   concat<KC, VC>(...iters: Array<Iterable<[KC, VC]> | PlainObjInput<KC, VC>>): KeyedSeq<K | KC, V | VC>;
                                            ^^^^^^^^^^^^^^^^^^ type application of identifier `Iterable`. See: node_modules/immutable/dist/immutable.js.flow:481
      Error:
       37:     .map(operation);
                    ^^^^^^^^^ function type. This type is incompatible with the expected param type of
      490:     mapper: (value: V, key: K, iter: this) => M,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ function type. See: node_modules/immutable/dist/immutable.js.flow:490
        This parameter is incompatible:
           36:     .concat(Map([[endKey, blockMap.get(endKey)]]))
                                         ^^^^^^^^^^^^^^^^^^^^ undefined. This type is incompatible with
           27:   operation: (block: ContentBlock) => ContentBlock,
                                    ^^^^^^^^^^^^ ContentBlock
      Member 2:
      481:   concat<KC, VC>(...iters: Array<Iterable<[KC, VC]> | PlainObjInput<KC, VC>>): KeyedSeq<K | KC, V | VC>;
                                                                 ^^^^^^^^^^^^^^^^^^^^^ type application of polymorphic type: PlainObjInput. See: node_modules/immutable/dist/immutable.js.flow:481
      Error:
       36:     .concat(Map([[endKey, blockMap.get(endKey)]]))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Map. This type is incompatible with
       33: type PlainObjInput<K, V> = {[key: K]: V, __proto__: null};
                                                               ^^^^ null. See: node_modules/immutable/dist/immutable.js.flow:33

I don’t actually know what the original intent is of passing an array of tuples to Map inside that concat. From what I can tell, the result is equivalent to the result of invoking Map({[endKey]: blockMap.get(endKey)}); my best guess is the array version is being used to avoid having to use ES2015 computed property names. However, when I change it to:
    .concat(Map({[endKey]: blockMap.get(endKey)}))

…the flow error goes away.

I think this is a legit error, but just the signal is buried in the noise.
Here's the signal:
   This parameter is incompatible:
           36:     .concat(Map([[endKey, blockMap.get(endKey)]]))
                                         ^^^^^^^^^^^^^^^^^^^^ undefined. This type is incompatible with
            27:   operation: (block: ContentBlock) => ContentBlock,
                                    ^^^^^^^^^^^^ ContentBlock

What's happening here is the following:

First, concat is actually doing the right thing: Map<K1, V1>.concat(Map<K2, V2>) ~> Map<K1|K2, V1|V2>`

And Map([[endKey, blockMap.get(endKey)]]) is being interpreted as Map<string, ContentBlock|void>

Because blockMap.get(endKey) returns ContentBlock|void

So the result of concat is also: Map<string, ContentBlock|void>

Which .map(operation) diligently forwards.

Where the value type ContentBlock|void isn't compatible with with the function argument operation: (block: ContentBlock) that does not accept undefined.

My suggestion for a fix would be to use something like nullthrows() around the result of .get() or some sort of invariant that the block at endKey definitely exists.
I'm not sure why the computed property names causes the error to go away - my guess there is that Flow has limited support for computed properties and misses the opportunity to check the type.
Part of why this error is tough to follow is because Flow is trying to show you that there are two possible ways of applying concat() (one for Iterable, one for PlainObj) and neither will work. Flow isn't really sure which you intended, so it shows you why both don't work, but because the clearly you intended on using the first and the punchline of the error is usually at the end of the string of Flow's rationale, and the second unindented case is printed right after it, the highest value bit gets lost in the middle of a blob of text.
/cc @calebmer for an example of error message that could be much better communicated
My suggestions for how errors like this could be improved:

Immediately after the where, explain why there is an inconsistency, then follow up with the how Flow arrived at that conclusion (upside down order).

When there are multiple Members, (or a nested tree of branching multiple members) Flow should have some idea of which one was most likely intended and start with that message. If there are alternative members that might have been intended, those should be shown next, and if there are any paths through member that are very unlikely to be intended in comparison to others, they can be omitted unless a check with --verbose is added.

Type arguments like "Type argument T is incompatible:" are not useful out of context. Which T? Where was that defined? When the error occurred, what was the actual type of T?

Use color and formatting in the terminal instead of ascii art

Finally, Flow should check the width of the terminal window before printing so there is an intelligible visual hierarchy to the error output to make it easier for the eye to follow.

(Some good advice on this from Elm: elm-lang.org/blog/compiler-errors-for-humans)
Here's a mockup of how I wish this error had been communicated to be easier to read:

Thanks for the detailed explanation! I found it very helpful to follow your logic as you traced the type logic, and using nullthrows took care of it.


iOS permissions not asked, and crash on ImagePicker being null

Bug
on android everything is working smooth, but in ios permissions are not asked even though I added the permissions to the info.plist. Then, when I try to open the image picker the app crashes with the exception of null pointer. ImagePicker.showImagePicker in this line ImagePicker is null..
Environment info
React native info output:
   React Native Environment Info:
    System:
      OS: macOS 10.14.4
      CPU: (4) x64 Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz
      Memory: 215.69 MB / 8.00 GB
      Shell: 3.2.57 - /bin/bash
    Binaries:
      Node: 8.11.2 - /usr/local/bin/node
      npm: 6.4.1 - /usr/local/bin/npm
      Watchman: 4.9.0 - /usr/local/bin/watchman
    SDKs:
      iOS SDK:
        Platforms: iOS 13.2, DriverKit 19.0, macOS 10.15, tvOS 13.2, watchOS 6.1
      Android SDK:
        API Levels: 26, 27, 28
        Build Tools: 22.0.1, 23.0.1, 24.0.2, 26.0.0, 26.0.1, 27.0.3, 28.0.1, 28.0.3
        System Images: android-22 | Google APIs Intel x86 Atom, android-24 | Google APIs Intel x86 Atom, android-24 | Google Play Intel x86 Atom
    IDEs:
      Xcode: 11.2.1/11B500 - /usr/bin/xcodebuild
    npmPackages:
      react: 16.6.3 => 16.6.3
      react-native: 0.59.9 => 0.59.9
    npmGlobalPackages:
      create-react-native-app: 1.0.0
      react-native-cli: 2.0.1
      react-native-git-upgrade: 0.2.7

Library version: 0.28.0
Steps To Reproduce

start a new project
install this library
call ImagePicker
run on ios 13
crash

Describe what you expected to happen:

Permissions should be asked
camera should open

Reproducible sample code
import ImagePicker from 'react-native-image-picker'; I inserted the library with this,
const options = {
      quality: 1.0,
      maxWidth: 500,
      maxHeight: 500,
      storageOptions: {
        skipBackup: true,
      },
      cancelButtonTitle: 'Kapat',
      takePhotoButtonTitle: 'Fotoğraf çek..',
      takePhotoButtonTitle: 'Fotoğraf çek..',
      chooseFromLibraryButtonTitle: 'Galeriden seç..',
      title: 'Seç'
    };

    ImagePicker.showImagePicker(options, response => {
      console.log('Response = ', response);

      if (response.didCancel) {
        console.log('User cancelled photo picker');
      } else if (response.error) {
        console.log('ImagePicker Error: ', response.error);
      } else if (response.customButton) {
        console.log('User tapped custom button: ', response.customButton);
      } else {
        this.uploadImage(response);

      }
    });

*** Terminating app due to uncaught exception 'RCTFatalException: Unhandled JS Exception: null is not an object (evaluating 'o.showImagePicker')', reason: 'Unhandled JS Exception: null is not an object (evaluating 'o.showImagePicker'), stack:
showImagePicker@809:546
value@940:2260
value@:

above is the exception I got

I just find out the problem.
react-native link did not link the library, so the mage picker lib did not build with the app. Thats why I as getting null pointer exception because there was no lib.
What I did is manually add the library.
If someone has encountered this problem try manually add the lib

@kyhnlbyrk so what is your solution for this because I found this bug on my App too.

My solution is add privacy in Info.plist
<key>NSCameraUsageDescription</key>
	<string>Access to take a photo by camera</string>
<key>NSAppleMusicUsageDescription</key>
	<string>Access to pick a photo</string>
<key>NSPhotoLibraryUsageDescription</key>
	<string>Access to pick a photo</string>

Correction :
import * as ImagePicker from 'react-native-image-picker';


Annotation - proper temporal part

From<EMAIL_ADDRESS>on August 03, 2012 05:37:02
The "has associated axiom(FOL)" annotations numbers 117-002 and 116-001 refer to "proper temporal part of" but are annotations of "temporal part of". Meanwhile, there is no annotations to elucidate the meaning of "proper temporal part of". Is this intended or are those annotations misplaced?
Original issue: http://code.google.com/p/bfo/issues/detail?id=118

From<EMAIL_ADDRESS>on August 13, 2012 21:57:12
Unintended. Fix is to correct the annotation target in the reference. Will do in next round of edits of reference.
Status: Accepted
Owner<EMAIL_ADDRESS>


[DOCS]  Update README.md

Issue Description
There are a few grammatical and spelling errors.

In the sentence "Before getting into it, it make you sure you have pnpm installed," it should be "make sure" instead of "it make sure."
In the sentence "Push your changes into your branch," it should be "Push your changes to your branch."

Additional Context
No response
Screenshots or Examples (if applicable)
No response
Proposed Solution (optional)
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue (optional)
[X] I am a GSSoC'23 contributor

sounds good, you may start working it


Cannot use TimeDistributed with hub.KerasLayer

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: None
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.2
Python version: 3.7
Bazel version (if compiling from source): None
GCC/Compiler version (if compiling from source): None
CUDA/cuDNN version: None
GPU model and memory: None

Describe the current behavior
When trying to apply the tf.keras.layers.TimeDistributed layer on top of a tensorflow_hub.KerasLayer, an exception (and not a useful one) is being raised
Describe the expected behavior
Just like any other tf.keras.layers.Layer, I'd expect one generated by the tensorflow_hub.KerasLayer to work
Standalone code to reproduce the issue
import os

import numpy as np
import tensorflow as tf
import tensorflow_hub as hub

os.environ["TFHUB_CACHE_DIR"] = '/tmp/tfhub'

# Create model
model = tf.keras.Sequential([tf.keras.layers.TimeDistributed(hub.KerasLayer("https://tfhub.dev/google/bit/s-r101x1/1",
                                                                            trainable=False)),
                             tf.keras.layers.LSTM(units=512),
                             tf.keras.layers.Dense(units=128,
                                                   activation=tf.nn.relu),
                             tf.keras.layers.Dense(units=1,
                                                   activation=tf.nn.sigmoid)])

model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
              metrics=['accuracy'])

# Fit
model.fit(x=np.random.randint(low=0, high=256, size=(1000, 10, 56, 56, 3)).astype(float),
          y=np.random.randint(low=0, high=2, size=(1000,)).astype(float), epochs=1)

Other info / logs Include any logs or source code that would be helpful to
Traceback:
2020-05-14 15:40:50.293457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-14 15:40:50.314782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 15:40:50.315236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:09:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.683GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-05-14 15:40:50.315361: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315403: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315438: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315474: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315508: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315542: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315577: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2020-05-14 15:40:50.315585: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-05-14 15:40:50.315827: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-14 15:40:50.337745: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency:<PHONE_NUMBER> Hz
2020-05-14 15:40:50.338512: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1b94000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-14 15:40:50.338545: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-14 15:40:50.341227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-14 15:40:50.341251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
Traceback (most recent call last):
File ".../site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 27, in
y=np.random.randint(low=0, high=2, size=(1000,)).astype(float), epochs=1)
File ".../site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
return method(self, *args, **kwargs)
File ".../site-packages/tensorflow/python/keras/engine/training.py", line 848, in fit
tmp_logs = train_function(iterator)
File ".../site-packages/tensorflow/python/eager/def_function.py", line 580, in call
result = self._call(*args, **kwds)
File ".../site-packages/tensorflow/python/eager/def_function.py", line 627, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File ".../site-packages/tensorflow/python/eager/def_function.py", line 506, in _initialize
*args, **kwds))
File ".../site-packages/tensorflow/python/eager/function.py", line 2446, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File ".../site-packages/tensorflow/python/eager/function.py", line 2777, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File ".../site-packages/tensorflow/python/eager/function.py", line 2667, in _create_graph_function
capture_by_value=self._capture_by_value),
File ".../site-packages/tensorflow/python/framework/func_graph.py", line 981, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File ".../site-packages/tensorflow/python/eager/def_function.py", line 441, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File ".../site-packages/tensorflow/python/framework/func_graph.py", line 968, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:
.../site-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
outputs = self.distribute_strategy.run(
.../site-packages/tensorflow/python/distribute/distribute_lib.py:951 run  **
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
.../site-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
.../site-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica
return fn(*args, **kwargs)
.../site-packages/tensorflow/python/keras/engine/training.py:531 train_step  **
y_pred = self(x, training=True)
.../site-packages/tensorflow/python/keras/engine/base_layer.py:927 call
outputs = call_fn(cast_inputs, *args, **kwargs)
.../site-packages/tensorflow/python/keras/engine/sequential.py:291 call
outputs = layer(inputs, **kwargs)
.../site-packages/tensorflow/python/keras/engine/base_layer.py:927 call
outputs = call_fn(cast_inputs, *args, **kwargs)
.../site-packages/tensorflow/python/keras/layers/wrappers.py:246 call
output_shape = self.compute_output_shape(input_shape).as_list()
.../site-packages/tensorflow/python/keras/layers/wrappers.py:190 compute_output_shape
child_output_shape = self.layer.compute_output_shape(child_input_shape)
.../site-packages/tensorflow/python/keras/engine/base_layer.py:699 compute_output_shape
raise NotImplementedError
NotImplementedError:

I have tried in colab with TF version 2.2 and was able to reproduce the issue.Please. find the gist here.Thanks!

From the error log, it seems that keras wrapper was expecting the tf_hub layer to implement the compute_output_shape() method, but it is not implemented. All the default keras layer has implemented compute_output_shape(), but I think it is missed by tf_hub.KerasLayer.
Adding @akhorlin from tf-hub side.

@gowthamkpr can we transfer this issue to tensorflow/hub?

Yes, please make this an issue in tensorflow/hub and assign it to me. hub.KerasLayer should supply its own .compute_output_shape but does not.
That said, I can get @ravikyram's colab to work by setting model.compile(..., run_eagerly=True) (and reducing the batch size from 1000 to 10). That's because default implementation inherited from the Layer base class for the eager case is good enough (as long as  the hub.KerasLayer(..., output_shape=...)override isn't used).

Ack, it seems that I somehow don't have the permission to transfer the issue. The dropdown list doesn't contain hub when I click "Transfer issue" on this page. In case this can't be done by any of you, feel free to recreate a new issue on tensorflow/hub.

Done. Created https://github.com/tensorflow/hub/issues/596, and closing this issue for now.

I had similar issue and this trick works for me
# Load KerasLayer
input = ...
layer = hub.KerasLayer("https://tfhub.dev/some/model/1",)
net  = layer(input)
net = tf.keras.Model(input , net)
# Add custom compute_output_shape
net.compute_output_shape = lambda x : (x[0],x[1],512)

# now use net to build my model
# Create model
model = tf.keras.Sequential([tf.keras.layers.TimeDistributed(net),...])

I had similar issue and this trick works for me
# Load KerasLayer
input = ...
layer = hub.KerasLayer("https://tfhub.dev/some/model/1",)
net  = layer(input)
net = tf.keras.Model(input , net)
# Add custom compute_output_shape
net.compute_output_shape = lambda x : (x[0],x[1],512)

# now use net to build my model
# Create model
model = tf.keras.Sequential([tf.keras.layers.TimeDistributed(net),...])

Greate ! Thanks !


HADOOP-18503. Upgrade Upgrade Huawei OBS client to <IP_ADDRESS>

Description of PR
Currently, the Huawei OBS client depends on OkHttp[1], we should upgrade it to the latest version[2] to get rid of OkHttp, to unblock removing OkHttp from Hadoop. Why remove OkHttp? Because the new OkHttp depends on Kotlin but Hadoop only uses it in a few places, replacing OkHttp w/ existing Apache Http client could avoid depending on Kotlin libs.
[1] https://mvnrepository.com/artifact/com.huaweicloud/esdk-obs-java/<IP_ADDRESS>
[2] https://mvnrepository.com/artifact/com.huaweicloud/esdk-obs-java/<IP_ADDRESS>
How was this patch tested?
Existing test.
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 52s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
41m 47s

trunk passed

+1 :green_heart:
compile
0m 32s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 31s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
0m 36s

trunk passed

+1 :green_heart:
javadoc
0m 38s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 30s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
69m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 27s

the patch passed

+1 :green_heart:
compile
0m 24s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

-1 :x:
javac
0m 24s
/results-compile-javac-hadoop-cloud-storage-project_hadoop-huaweicloud-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt
hadoop-cloud-storage-project_hadoop-huaweicloud-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)

+1 :green_heart:
compile
0m 24s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

-1 :x:
javac
0m 24s
/results-compile-javac-hadoop-cloud-storage-project_hadoop-huaweicloud-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt
hadoop-cloud-storage-project_hadoop-huaweicloud-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 25s

the patch passed

+1 :green_heart:
javadoc
0m 24s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
26m 16s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 26s

hadoop-huaweicloud in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

100m 48s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4989

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint

uname
Linux 575cc38c8b2d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 3ce0cef5898351ebc43daa36654070ff39784399

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/1/testReport/

Max. process+thread count
531 (vs. ulimit of 5500)

modules
C: hadoop-cloud-storage-project/hadoop-huaweicloud U: hadoop-cloud-storage-project/hadoop-huaweicloud

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I'm afraid there are no alternatives for the deprecated constructors, so I leave it as-is.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 55s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
42m 31s

trunk passed

+1 :green_heart:
compile
0m 32s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 31s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 34s

trunk passed

+1 :green_heart:
mvnsite
0m 36s

trunk passed

+1 :green_heart:
javadoc
0m 39s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 31s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m  4s

trunk passed

+1 :green_heart:
shadedclient
23m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 26s

the patch passed

+1 :green_heart:
compile
0m 22s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

-1 :x:
javac
0m 22s
/results-compile-javac-hadoop-cloud-storage-project_hadoop-huaweicloud-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt
hadoop-cloud-storage-project_hadoop-huaweicloud-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

+1 :green_heart:
compile
0m 21s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

-1 :x:
javac
0m 21s
/results-compile-javac-hadoop-cloud-storage-project_hadoop-huaweicloud-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt
hadoop-cloud-storage-project_hadoop-huaweicloud-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 17s

the patch passed

+1 :green_heart:
mvnsite
0m 23s

the patch passed

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
0m 48s

the patch passed

+1 :green_heart:
shadedclient
23m  7s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 25s

hadoop-huaweicloud in the patch passed.

+1 :green_heart:
asflicense
0m 41s

The patch does not generate ASF License warnings.

100m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4989

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle

uname
Linux 6c42c4f7303b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 97ddda42f44b1fc1adbc82bf7e4ef91e2ad8a535

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/2/testReport/

Max. process+thread count
597 (vs. ulimit of 5500)

modules
C: hadoop-cloud-storage-project/hadoop-huaweicloud U: hadoop-cloud-storage-project/hadoop-huaweicloud

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4989/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

I don't have OBS account, cc @Yikun, could you please help to run integration tests?

kindly ping @hililiwei as I see you are working OBS integration for iceberg

kindly ping @hililiwei as I see you are working OBS integration for iceberg

@pan3793 thank you. I'll keep an eye on it.

moved to a hadoop jira (HADOOP-18503)
needs the checkboxes on the top filled in

license review, including transitive dependencies
license-binary update
someone to run the integration tests

No OBS account, abandoned.


Get-Whiteboard -UserId throws gateway error

[Enter feedback here]
In an attempt to find out whiteboard usage in our tenant, i was getting whiteboard for all users and it errored out in the middle with gateway error. I wish there is a flag to return all whiteboards in the tenant and corresponding owners.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c8a6935c-013b-7196-1f41-434c4486d382
Version Independent ID: 22971c35-eeda-ee3a-4cd8-2df1af8f43f2
Content: Get-Whiteboard (MicrosoftWhiteboardAdmin)
Content Source: whiteboard/whiteboard-ps/whiteboard/Get-Whiteboard.md
Service: whiteboard-powershell
GitHub Login: @SerdarSoysal
Microsoft Alias: serdars

Hello @vellabati how are you? Hope you are doing good.
Here, we are limited to documentation changes and fixes, is there any special fix you see to documentation?
Otherwise, and please corrent me if I am wrong, your inquiry looks more suitable to uservoice, please follow here for directions https://support.microsoft.com/en-us/topic/uservoice-pages-430e1a78-e016-472a-a10f-dc2a3df3450a and https://whiteboard.uservoice.com/
Thank you

@vellabati  Thank you very much for your kind feedback. Considering your valuable feedback, as advised please log this issue as a User Voice. Concerning team will evaluate and take the appropriate action, as per their schedule. We proceed here to close it. Thanks again for taking out some time and sharing your feedback.


test 1.2 (updates to objects) failing

TEST 1.0 Validate CVE -> CWE relationship (cve-cwe) WORKS
python3 -m unittest tests/test_01_00_cve_cwe.py

Contains 3 CWE refs.
TEST 1.1 Validate CVE -> CWE relationship (cve-cwe) WORKS
Test 1.0 must be run.
CVE-2019-16278 has CWE-404 added to it
python3 -m unittest tests/test_01_01_cve_cwe_update_1.py

TEST 1.2 Validate CVE -> CWE relationship (cve-cwe) FAILS
Test 1.1 must be run.
All CWE references are removed from CVE-2019-16278
python3 -m unittest tests/test_01_02_cve_cwe_update_2.py

Note on logic
This logic works in ACTIP, see this test
TEST 1.3 Perform ANOTHER update to change CAPEC Attack Pattern -> ATT&CK Attack Pattern relationship (capec-attack)
Note, test 1.2 must have been run for this test to work.
This time we remove 2 of the ATT&CK references inside the CAPEC object (attack-pattern--897a5506-45bb-4f6f-96e7-55f4c0b9021a), with 2 remaining T1040 and T1111 (total 4 ATT&CK links). T1650 and T1574.010 are removed (total 2 ATT&CK links). This is now the same as the original stix-capec-v3.9.json object
python3 -m unittest tests/test_01_3_capec_to_attack.py

is this test passing for you @fqrious ?
======arango_cve_processor run successfully======
.FF
======================================================================
FAIL: test_02_arango_cve_processor_note_new (tests.test_01_02_cve_cwe_update_2.TestArangoDB.test_02_arango_cve_processor_note_new)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/arango_cve_processor/tests/test_01_02_cve_cwe_update_2.py", line 76, in test_02_arango_cve_processor_note_new
    self.assertEqual(result_count, [1], f"Expected 1 documents, but found {result_count}.")
AssertionError: Lists differ: [4] != [1]

First differing element 0:
4
1

- [4]
+ [1] : Expected 1 documents, but found [4].

======================================================================
FAIL: test_03_arango_cve_processor_note_new (tests.test_01_02_cve_cwe_update_2.TestArangoDB.test_03_arango_cve_processor_note_new)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dgreenwood/Documents/repos/dogesec/arango_cve_processor/tests/test_01_02_cve_cwe_update_2.py", line 92, in test_03_arango_cve_processor_note_new
    self.assertEqual(result_count, [5], f"Expected 5 documents, but found {result_count}.")
AssertionError: Lists differ: [2] != [5]

First differing element 0:
2
5

- [2]
+ [5] : Expected 5 documents, but found [2].

----------------------------------------------------------------------
Ran 3 tests in 1.307s

FAILED (failures=2)
``


[SPARK-17903][SQL] MetastoreRelation should talk to external catalog instead of hive client

What changes were proposed in this pull request?
HiveExternalCatalog should be the only interface to talk to the hive metastore. In MetastoreRelation we can just use ExternalCatalog instead of HiveClient to interact with hive metastore,  and add missing API in ExternalCatalog.
How was this patch tested?
existing tests.

cc @yhuai @gatorsmile

Test build #66882 has started for PR 15460 at commit 79ec4fe.

Test build #66882 has finished for PR 15460 at commit 79ec4fe.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66882/
Test FAILed.

Merged build finished. Test FAILed.

Test build #66893 has started for PR 15460 at commit f3ba6e0.

Test build #66893 has finished for PR 15460 at commit f3ba6e0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66893/
Test FAILed.

Merged build finished.

LGTM if tests pass :)

Caused by: sbt.ForkMain$ForkError: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'default' not found in database 'srcpart'; Seems we swapped the db and table name somehow.

Test build #66922 has started for PR 15460 at commit 628fd05.

LGTM except one comment about test case

Test build #66922 has finished for PR 15460 at commit 628fd05.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66922/
Test PASSed.

Test build #66948 has started for PR 15460 at commit 82df987.

Test build #66948 has finished for PR 15460 at commit 82df987.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

thanks for the review, merging to master!


Put confirmation of "id not being a medical record" in the validation step only once.

the confirmation has been removed from the medical action section. I don't see it anywhere except when validating a newly created phenopacket


Can't use docker stop with tomcat image. Exit code 143 is returned. Exit code 0 was expected

Hi,
I played around with the docker container using the docker commands docker start/stop/pause... I'd like to use the docker restart policy with the option on-failure:

on-failure[:max-retries] Restart only if the container exits with a non-zero exit status. Optionally, limit the number of restart retries the Docker daemon attempts.

The tomcat docker container does not stops in the expected way. I expected exit code 0 - but the container stops with exit code 143.
vagrant@development:~$ docker --version
Docker version 1.12.3, build 6b644ec
vagrant@development:~$ docker run -d --name mytomcat tomcat:8.5.8
00eca311039ba413b1dcfb59b0e64551e5460a4af4777057bf1476bc46e84864
vagrant@development:~$ docker stop mytomcat
mytomcat
vagrant@development:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                       PORTS               NAMES
00eca311039b        tomcat:8.5.8        "catalina.sh run"   9 seconds ago       Exited (143) 2 seconds ago                       mytomcat

Possible solution:

do not execute catalina.sh run. Use a wrapper script instead which execute catalina.sh start.
The wrapper script could handle the unix signals using trap as described in trapping-signals-in-docker-containers or in gracefully-stopping-docker-containers/

What do you think about this? Could this be a working solution?
Regards
@cpauls

Did you check the logs from the container to see why it's failing?  (docker logs mytomcat)

Hi,
the logfiles are looking good.
vagrant@development:~$ docker run -d --name mytomcat tomcat:8.5.8
e2cdfe3c31612b515103958b488c5265736b4530580cc13a4ef64ef233bc4d0e
vagrant@development:~$ docker stop mytomcat
mytomcat
vagrant@development:~$ docker logs mytomcat
12-Dec-2016 09:07:44.868 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.8
12-Dec-2016 09:07:44.871 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Nov 3 2016 21:14:13 UTC
....
12-Dec-2016 09:07:46.271 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1195 ms
12-Dec-2016 09:07:48.460 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
12-Dec-2016 09:07:48.522 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
12-Dec-2016 09:07:48.573 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
12-Dec-2016 09:07:48.617 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
12-Dec-2016 09:07:48.618 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
12-Dec-2016 09:07:48.619 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
12-Dec-2016 09:07:48.620 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
vagrant@development:~$

I think the problem is executing catalina.sh directly.  For starting the docker image the command CMD ["catalina.sh", "run"] is executed.
You can reproduce this by hand.
First I start a docker the tomcat directly in the dockercontainer:
vagrant@development:~$ docker run -ti --rm  --name playground tomcat:8.5.8 /bin/bash
root@27378401fcac:/usr/local/tomcat# /usr/local/tomcat/bin/catalina.sh run
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/lib/jvm/java-8-openjdk-amd64/jre
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
12-Dec-2016 09:53:49.114 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.8
...
12-Dec-2016 09:53:49.355 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.8
12-Dec-2016 09:53:49.373 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/ROOT

The tomcat is now running. Now I  enter the container directly in a second shell and kill the tomcat process with SIGTERM like docker does.
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
27378401fcac        tomcat:8.5.8        "/bin/bash"         4 minutes ago       Up 3 minutes        8080/tcp            playground
vagrant@development:~$ docker exec -ti playground ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:53 ?        00:00:00 /bin/bash
root         5     1  1 09:53 ?        00:00:02 /usr/lib/jvm/java-8-openjdk-amd6
root        31     0  0 09:57 ?        00:00:00 ps -ef
vagrant@development:~$ docker exec -ti playground kill -15 5
TERM
vagrant@development:~$

In the first shell the tomcat is stopping. I can ask for the exit code with echo $?. The exitcode is 143
12-Dec-2016 09:53:49.355 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.8
12-Dec-2016 09:53:49.373 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/ROOT
12-Dec-2016 09:55:26.908 INFO [localhost-startStop-1] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [97,094] milliseconds.
12-Dec-2016 09:55:26.941 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/ROOT has finished in 97,568 ms
12-Dec-2016 09:55:26.941 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/host-manager
12-Dec-2016 09:55:27.012 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/host-manager has finished in 71 ms
12-Dec-2016 09:55:27.013 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/manager
12-Dec-2016 09:55:27.057 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/manager has finished in 44 ms
12-Dec-2016 09:55:27.058 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/examples
12-Dec-2016 09:55:27.468 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/examples has finished in 410 ms
12-Dec-2016 09:55:27.469 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory /usr/local/tomcat/webapps/docs
12-Dec-2016 09:55:27.497 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/docs has finished in 28 ms
12-Dec-2016 09:55:27.507 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [http-nio-8080]
12-Dec-2016 09:55:27.522 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler [ajp-nio-8009]
12-Dec-2016 09:55:27.530 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 98222 ms
12-Dec-2016 10:01:45.220 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
12-Dec-2016 10:01:45.234 INFO [Thread-5] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["ajp-nio-8009"]
12-Dec-2016 10:01:45.235 INFO [Thread-5] org.apache.catalina.core.StandardService.stopInternal Stopping service Catalina
12-Dec-2016 10:01:45.290 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
12-Dec-2016 10:01:45.291 INFO [Thread-5] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-8009"]
12-Dec-2016 10:01:45.292 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
12-Dec-2016 10:01:45.294 INFO [Thread-5] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-8009"]
root@27378401fcac:/usr/local/tomcat# echo $?
143
root@27378401fcac:/usr/local/tomcat#

So back to the Possible solutions:

Tomcat itself has to fix it - I think the won't do that :simple_smile:
develop a wrapper script as suggested in my first post

Arg, I see -- I've been researching a little, and it seems like there might not actually be a nice way for Tomcat upstream to fix this issue.
http://stackoverflow.com/a/4192488/433558: exit code 143 corresponds to SIGTERM
http://stackoverflow.com/a/2541618/433558: Java applications can respond to the shutdown that's initiated by SIGTERM, but it does not appear that they can distinguish it from SIGINT or modify the resulting exit code in any sane way :disappointed:
:sob:

In my opinion this is not an issue, everything works as described
docker stop does exactly as the documentation states:
... a SIGTERM is sent to the main proces. ...

From the docker ps -a you are informed that indeed SIGTERM (143) has resulted in stopping the tomcat container.
Further notice: It may also have been a SIGKILL (137) if tomcat would have used more time then the grace period (default: 10 seconds) to shut down.
In the description of the run command with regard to the restart-policies it is stated that
....and so on until either the on-failure limit is hit, or when you docker stop or docker rm -f the container. ...

So when you give the stop command it will NOT be restarted.
What could be wrong is that giving the SIGTERM internally (without docker stop) one would expect the container to be restarted in this setup. But that may depend on the docker implementation...
I further tested the case when you send a different signal (e.g. kill -1 1). In that case the container is restarted as i would expect it to...
@cpauls can you describe what you wish to accomplish?
Note that i have a setup usig systemd (on CoreOS) in which i specified in the unit file to restart the tomcat container in this case (Restart=on-success and SuccessExitStatus=143). But that may not be an option in your case.

Closing, given that this appears either resolved or working appropriately. :+1:


Calling an NamedTuple#to_h on an empty now has meaningful error message

switched from making it return {} => Nil to just improving the compile-time error message

@karlseguin do you have any scenario for this?
I am not convinced the Symbol => Nil will be comfortable when you potentially will expect a hash of other type if the tuple is not empty. NamedTuple#to_a for example return Array(Tuple(NoReturn, NoReturn)).
Tuple does not have a #to_a but if it had, the same problem will be present there. You will sometimes have a Array(Nil) or and Array(T1 | T2 | ...). So, you will still be needing some case over time.
Another alternative could be to do something as follows. Depending on your use case it might be useful or not.
def some_key?(**opt)
  if opt.empty?
    :empty
  else
    opt.each do
      return opt.keys[0]
    end
  end
end

some_key? foo: 1 # => :foo
some_key?        # => :empty

I'm sure I could come up with a different (better) way to solve my specific problem. But it seems weird to me that calling a method on an instance would give a compiler error based on the state of that instance. On the other hand, I guess the fact the they're known at compile time means that we can do the check at compile time.
I definitely think the current behavior is wrong (if for no other reason as being an unhelpful error message), so, we could:
1 - Change the default type to something else, but what?
2 - Raise a exception
3 - Improve the compile-time error message
My case was factories for creating tests objects that looked like:
def user(args = NamedTuple.new())
   # somewhere args.to_h was being called

the simplest solution would be to make it nullable.

For reference, this is how it's currently being handled
in /opt/crystal/src/named_tuple.cr:402: macro didn't expand to a valid program, it expanded to:

================================================================================
--------------------------------------------------------------------------------
   1.
   2.       {
   3.
   4.       }
   5.
--------------------------------------------------------------------------------
Syntax error in expanded macro: macro_4610676704:2: for empty hashes use '{} of KeyType => ValueType'

      {
      ^

================================================================================

    {% begin %}

Why not make it return nil. Because this is done via macros, I don't think that you will actually have to deal with to_h returning nil until there's an empty tuple in the union you're calling to_h on.

I changed it to give a more meaningful compile-time error. This seems like the least risky improvement. I couldn't figure out how to get it under test though.

👍  for returning nil.
Otherwise scenario below becomes impossible:
def foo(**options)
  options.to_h
end

to_h returns hash, not nil. Original PR was OK

I still think that raising in compile time is the right thing to do.
Dealing with a hash version of a named tuple is far from a type safe thing.
If the developer want to to something with the hash coming from options.to_h:
def foo(**options)
  # use options.to_h
end

there should be some basic assumptions of the types of the values. for example: all integers
def foo(**options)
  options.to_h.values.sum
end

So foo(a: 1, b: 2) # => 3 works. And foo(a: 1, b: "2") fails to compile.
foo also fails to, before this PR with a more cryptic error message.
So failing with a clear "Can't convert an empty NamedTuple to a Hash" is an improvement.
What happens if we want to return a hash (of some type) for the empty tuple?
For hash type we choose Symbol => Nil, NoReturn => NoReturn, etc there will be a case that fails to compile when the original named tuple is empty due to a type mismatch between the hash for the empty tuple ad the hash for the non empty tuple.
So, in the sample case of returning the sum of the values of the named tuple the programer is left with two options.
a) overload:
def foo
  0
end

def foo(**options)
  options.to_h.values.sum
end

b) split the case with and weird is_a? (I think there is no other way to express the empty tuple type).
def foo(**options)
  if options.is_a?(typeof(NamedTuple.new))
    0
  else
    options.to_h.values.sum
  end
end

@karlseguin I authored the changeset, there was an unneeded begin/end block since now there is a macro if. After CI pass this should be ready to merge.
I also added a spec for this (yet upon #2391 it could be moved to the std)


Fixes for wallcreationModal and map view

Added check for login for wallcreationModal,
and missing location fields to the mongodb schema
fixes #18.

Merging as checked successfully


When initialising groups for DDP, a certain amount of memory is accumulated on the 0th GPU.

Bug description
Hi, I want to use DDP in pytorch lightning, but I'm reaching out to you because I saw a strange error today.
As shown in the code below, when I initialise the model, it keeps taking up more than a certain amount of memory. If this memory grab was happening on all GPUs, I'd be fine with it, but you can see that it's consistently piling up to zero.
    data = DataModule(args.batch_size)

    checkpoint_callback = ModelCheckpoint(
        dirpath = args.checkpoint_dir,
        filename = '{epoch}-{val_loss:.2f}',
        monitor="val_loss"
    )

    trainer = Trainer(
        log_every_n_steps=50,
        accelerator='gpu',
        strategy = DDPStrategy(find_unused_parameters=False),
        devices=4,
        max_epochs=args.max_epochs,
        resume_from_checkpoint=args.resume_from_checkpoint,
        logger=wandb_logger,
        enable_progress_bar=True,
        callbacks=[
            checkpoint_callback,
        ]
    )
    model = CLIP() # that I implemented

    trainer.fit(model, data)

Additionally, if you pass the DDPStrategy class or the string "ddp" as the strategy argument to Trainer, you can see that a certain amount of memory continues to be stacked at time 0 whenever group is initialised, as shown below.

I didn't get this error before, but it happened suddenly and I'm very confused. Maybe I'm missing something, and if anyone has experienced it, please share. Thank you.
What version are you seeing the problem on?
v1.8
How to reproduce the bug
.

Error messages and logs
There was no error message but only memory imbalance.
 Note that if you do, and the checkpoint contains cuda tensors, they will be allocated on device 0. You will have to add map_location="cpu" if you want to prevent that. Or you may have some code before trainer.fit() that allocates device memory. This can't be controlled by Lightning.

Hey @drizzle0171
Can you check my reply please?
Have you tried map_location="cpu" when loading the checkpoint?

Oh! I'm sorry I missed your message. As you said, I loaded a checkpoint from a library. So I fixed the problems. Thank you very much.


MudDateRangePicker runs into NullPoint exception

Bug type
Component
Component name
MudDateRangePicker
What happened?
If editing is on (Editable="true") and the user manually deletes both dates the Components runs into a null point exception.
Expected behavior
The range would be set to an empty range
Reproduction link
https://mudblazor.com/components/datepicker#range-picker-usage
Reproduction steps

In the editable DateRangePicker (Basic range picker (editable)) delete both dates
Unselect the component
Site crashes

Relevant log output
System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=MudBlazor
  StackTrace:
   at MudBlazor.MudDateRangePicker.set_RangeText(Range`1 value)

value is null so value.Start and value.End are not possible

Version (bug)
5.1.2
Version (working)
No response
What browsers are you seeing the problem on?
Firefox, Chrome
On what operating system are you experiencing the issue?
Windows
Pull Request

[x] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@JonBunator I can try to work on a fix, unless of course if @matthbr wants to do it.


Added .travis.yml

Implements #7

Could you add a status badge to README right below the Ruby.tw Reboot header? Thanks!

@rubytaiwan/rubyists Please take a look! Leave :+1: or LGTM if you think this Pull Request is good to go.

I've committed your changes to master. Thanks for your contributions :+1:


feat(chisel): improve inspector

Motivation
Currently many expressions are missing from the Type enum which handles expressions to be inspected
Fixes #4130
TLDR:

Can inspect all Solidity expressions, excluding tuples, with the correct type. This includes, but not limited to:

All global variables, with correct abi.decode, type(T).max etc types
All type associated methods (array.length, function.address)
All assignments, which also get added to the final code

Inspected arithmetic operations will return int if at least 1 of the operands is also an int

Solution

Add missing expressions
Properly handle functions, accesses and tuples
Refactor global variable logic to another method
Clean up 73 levels of indentation functions

Listed in the code

Continue inspecting if inspectoor variable fails

Any tuple (or empty return) fails compilation due to it not being able to be encoded in inspectoor
This is getting already too big, I think this is better for another PR

[x] Encode array length and relative offset for dynamic arrays Array literals as Type::FixedArray
[x] If assignment expressions return Some and get "inspected", the assignment does not happen
[x] Exhaustively match on from_expression (blocked by the above)
[x] Use Int(n) for arithmetic operations

Might not be needed or feasible:

(?) Rational numbers (no corresponding ParamType ?)

Not sure how I am supposed to configure Solc to run tests ..

code-wise this lgtm, not too familiar with the issue, can we get a test case for this perhaps?
wdyt @clabby @rkrasiuk

Busy with some work stuff, but will review this & #4147 later on tonight or tomorrow morning! Appreciate the improvements @DaniPopes 🚀

same as what @mattsse said, the code looks good. deferring to @clabby for approval

code-wise this lgtm, not too familiar with the issue, can we get a test case for this perhaps?
wdyt @clabby @rkrasiuk

The initial impetus for this PR was https://github.com/foundry-rs/foundry/issues/4130 which includes some failing test cases pertaining to signed integers.

we merged two other chisel PRs yesterday,
just to rule it out, this does not panic with nightly chisel?

On my machine, it crashes every time a variable or expression is inspected (regardless of the expression type)

I introduced infinite recursion in 799e2b42653fa88b27d9bb92e404b430dfc3611f by mistake because I didn't merge the try_as_ethabi match arms properly. Fixed in ed8683863ba6d5a24659241bd8cc0699bdf48367, sorry about that


Download server down.

The download page at https://www.minecraft.net/en-us/download/server/bedrock/ is broken and missing download links.
(not your issue).
I got around it by specifying the VERSION=1.16 which I had already downloaded. But it would be good to have an option for skip download and just use last downloaded version...

The latest image will now use the existing bedrock_server-*, if present, when the version lookup fails. It seemed reasonable to have it always fallback like that rather than making it an option.

Thanks


Listener for previous and next month

Hi there Kalendar Team!!
First of all I wanna say Congrats and thanks for this incredible library.
I Implemented it on my personal project and it works great. However there is a feature that I'd like you to add. I'm talking about the onPreviousClick and onNextClick on KalendarHeader. I'd like to know when user clicks on those options. You could return a lambda to let us to handle the logic.
Thanks in advance guys!

Hey buddy, I am about to add a new design version for it and there you can add your own custom design as you like

Hey mind checking
https://github.com/hi-manshu/Kalendar/releases/tag/v1.3.0
It has options to add own custom Header/ Please do check

Hey!... I'll check it. Thanks


Move encoding of Component-Specific types and sections to wasm-tools

From this discussion it's clear that there is an existing effort to add this functionality for modules into wasm-tools and the wasm_encoder itself.
Moving this functionality and submitting a PR there by modifying the Reencode trait will make maintainability much better and clean up our code.

PR has been opened here


Grouping Issue

I am having an issue where I have 4 creatures all from the same actor but they do not group properly. I have 3 in a group and one on his own. But they are from the same exact actor.

Was the Assassin(2) added after rolling initiative? Or was it there from the get-go?

@Avitale34 this seems to happen if you add a token to combat after rolling initiative. I've made a fix that I'll release this week still, to rather add the token to the already created group using that group's initiative.


Refactor for macro import sanity

functionally I just moved some things deeper into the library so I can refer to them from the builtins crate as I use the proc-macro. This brings us our first function str-sub.
basically in order to use macros you have to have fully qualified imports. Currently I'm using the macro in the builtins crate. but we could use it with any crate that imports compile_state and vm. Either way I had an issue where the macro was using the imports builtins::add_builtin for instance. And it would work in the tests trybuild tests, because those are their own binaries and compile against the public API and hence, see the builtins crate that way. But when I use the macro inside the builtins crate it exepcts the import to be crate::add_builtins. Anyway, let me know if we should put these elsewhere but it would be very handy if we could keep them in a dependency of builtins so I can both use and test them in there.

I think I would rather see the stuff you moved into compile_state in it's own crate. I don't really feel it belongs, what was you thinking in putting it here? It does not look like you touched CompileEnvironment which kinda why compile_state exists (to create a SloshVM).

totally get it. wasn't sure where to put them so I just put it in the minimal dependency. Created a new crate in the workspace. I think this is better.


Commands that require the level name do not work, support for & with gradient and hexadecimal colors is missing.

Hi, @CPJiNan I was testing the version you gave me in this issue #5 and when I put the command /akarilevel exp add   1000 and the same with the level add command, it didn't work and gave me these errors. -> https://paste.helpch.at/dabicaralo.rb
I think you forgot that I also asked you that for the levels you can also use & together with gradient and hexadecimal colors instead of §. (this I could not test if you added support or not since I did not work the commands).

Hi, @Zarkness
Thank you for helping with the testing!
The issue you encountered might be due to the Kotlin serialization dependency not loading. Please try restarting your server.
For some unknown reason, the plugin does not automatically enable the dependencies after the initial download. I am currently working on a fix for this.
Regarding hex colors, the colored() function in Taboolib seems to support this feature. This means you can directly use hex color codes.
For example:
{255-255-255} — RGB code
{255,255,255} — RGB code
{#FFFFFF} — HEX code
{BLUE} — Known color (English)
{蓝} — Known color (Chinese)

Hi, @Zarkness
Thank you for helping with the testing!
The issue you encountered might be due to the Kotlin serialization dependency not loading. Please try restarting your server.
For some unknown reason, the plugin does not automatically enable the dependencies after the initial download. I am currently working on a fix for this.
Regarding hex colors, the colored() function in Taboolib seems to support this feature. This means you can directly use hex color codes like &#FFFF00.

Hi, @Zarkness
Thank you for helping with the testing!
The issue you encountered might be due to the Kotlin serialization dependency not loading. Please try this build.
Regarding hex colors, the colored() function in Taboolib seems to support this feature. This means you can directly use hex color codes like &{#FFFF00}.

Hi, @Zarkness
Thank you for helping with the testing!
The issue you encountered might be due to the Kotlin serialization dependency not loading. Please try restarting your server. For some unknown reason, the plugin does not automatically enable the dependencies after the initial download. I am currently working on a fix for this.
Regarding hex colors, the colored() function in Taboolib seems to support this feature. This means you can directly use hex color codes.
For example:
{255-255-255} — RGB code {255,255,255} — RGB code {#FFFFFF} — HEX code {BLUE} — Known color (English) {蓝} — Known color (Chinese)

With the version that you passed me I already tried restarting the server several times and it is still the same, I did not try with the most recent one.
the colors could not you add support for the format that is used more globally? is that having on one side &b on another <blue> and another {blue} or for gradients {255-255-255} when normally it is usually <#7946FD>Text</#9B76FB> or <gradient:#7946FD:#9B76FB>Text</gradient>, is something ugly.

now it works without any problem and the color theme at least in the level action does not work.

the akarilevel levelup command does not work with or without op having the necessary permissions.
When I put as before to level 20 a condition now instead of raising me by much xp that I give the user does not rise to level 20 even if you have the permission remains at level 19, this is the config that I have.
I'm not going to bad but I'm getting a little tired of encountering problems with the plugin every now and then.
# 示例等级组
Example:
  # 展示名称
  Display: "示例等级组"
  # 来源设置
  Source:
    # 监听的经验来源
    # - "VANILLA_EXP_CHANGE"
    Subscribe:
      - "MYTHICMOBS_DROP_EXP"
  # 追踪设置
  # 当玩家追踪某个等级组时，将在原版经验条上显示其经验进度
  Trace:
    # 该等级组是否能被追踪
    Enable: true
    # 追踪该等级组所需条件 (Kether支持, 每行返回值均为 true 时才可追踪)
    # Condition:
    #   - false
    Condition: [ ]
    # 追踪该等级组后执行的动作 (Kether支持)
    Action: [ ]
  # 等级设置
  Level:
    # 最高等级
    Max: 100
    # 满足要求自动升级
    Auto-Levelup: true
    # 达到最大等级后不再获取经验
    Exp-Limit: true
    # 关键等级设置
    # 只需对关键等级进行修改, 处于非关键等级时将自动读取上一个关键等级的信息
    Key:
      # 0 级等级设置
      0:
        # 等级名称
        # %level% -> 对应等级
        Name: "§8Lv.§b0"
        # 升至该等级所需经验 (JavaScript支持)
        # %level% -> 对应等级
        Exp: "0"
        # 升至该等级所需条件 (Kether支持, 每行返回值均为 true 时才可升级)
        # Kether动作列表 https://kether.tabooproject.org/list.html
        # Condition:
        #   - "permission akarilevel.default"
        Condition: [ ]
        # 升级后执行的动作 (Kether支持)
        # Action:
        #   - command inline "give {{sender}} apple 1" as op
        Action: [ ]
      # 1 级 - 49 级(下一个关键等级之前的等级) 等级设置
      1:
        Name: "&{#4C88E4}Lv.§b%level%"
        Exp: "%level% * 5 + 100"
        Condition: [ ]
        Action:
          - title "&{#4C88E4}§lLevel UP!"
          - subtitle inline "{{ papi %akarilevel_lastlevelname% }} §7→ {{ papi %akarilevel_levelname% }}" by 10 20 10
          - actionbar inline "&{#98F5F9}[ §7{{ papi %akarilevel_exp% }} §8/ §7{{ papi %akarilevel_nextlevelexp% }} §8]"
      20:
        Name: "&{#4C88E4}Lv.§b%level%"
        Exp: "%level% * 5 + 100"
        Condition:
          - "permission testing.level.20"
        Action:
          - title "&{#4C88E4}§lLevel UP!"
          - subtitle inline "{{ papi %akarilevel_lastlevelname% }} §7→ {{ papi %akarilevel_levelname% }}" by 10 20 10
          - actionbar inline "&{#98F5F9}[ §7{{ papi %akarilevel_exp% }} §8/ §7{{ papi %akarilevel_nextlevelexp% }} §8]"
      # 50 级 - 100 级(满级) 等级设置
      50:
        Name: "§8Lv.§b%level%"
        Exp: "%level% * 10 + 100"
        Condition: [ ]
        Action:
          - title "§e§lLevel UP!"
          - subtitle inline "{{ papi %akarilevel_lastlevelname% }} §7→ {{ papi %akarilevel_levelname% }}" by 10 20 10
          - actionbar inline "§8[ §7{{ papi %akarilevel_exp% }} §8/ §7{{ papi %akarilevel_nextlevelexp% }} §8]"

I apologize for the inconvenience you have experienced. I am working on fixing the known issues and optimizing the plugin.
Additionally, here is an explanation of the Kether action list regarding hex color code statements. You don't need to rush to test it. Please stay tuned for the release of version 4.0.

Thank you for your patience.

I apologize for the inconvenience you have experienced. I am working on fixing the known issues and optimizing the plugin.
Please stay tuned for the release of version 4.0.
Additionally, here are the relevant descriptions in the Kether action list regarding hexcolor codes.

Thank you for your patience.

I apologize for the inconvenience you have experienced. I am working on fixing the known issues and optimizing the plugin.
Please stay tuned for the release of version 4.0.
Additionally, here are the relevant descriptions in the Kether action list regarding hexcolor codes. You don't need to rush to test it, please wait for the upcoming updates.

title inline "{{ color '&{#4C88E4}Hex&{#98F5F9}Color' }}" by 10 20 10
Thank you for your patience.

Additionally, the issue you mentioned regarding the levelup command not working has been fixed.

Language en_US are supported in this build.

Language en_US is supported in this build.
Regarding the level condition issue, it is still pending a fix.

it's ok, when the devs are so nice and good people I have no problem to wait for them to finish fixing things.
Ok, now I try the last possible snapshot, thanks for fixing all the bugs I report you

one thing, shouldn't the akarilevel levelup command ask you for the name of the experience system?

by the way, I am testing the last snapshot that is this https://github.com/CPJiNan/AkariLevel/actions/runs/9630761802 and when I do levelup or I have the autolevelup activated I don't get the title, besides it doesn't generate the 2 level files, it only generates 1.

by the way, I am testing the last snapshot that is this https://github.com/CPJiNan/AkariLevel/actions/runs/9630761802 and when I do levelup or I have the autolevelup activated I don't get the title, besides it doesn't generate the 2 level files, it only generates 1.

This issue has been fixed, and the generation of level file depends on your language.
Additionally, the feature request regarding the levelup command of specific level group will be adopted.
Besides, the snapshot version may have some problems, so please refer to the official version 4.0.

Just now I pushed an update, which may be the official version 4.0 update.
During my testing, it seems that all the issues you mentioned have been fixed. If you think it's ok, I will upload it to the SpigotMC update.
Thank you for your help with AkariLevel testing so far.

ok, so far for what I've tested if this well and works perfectly the only thing the language as it is put as the level I always get the Chinese version, when I use the English language, should not put the 2 default if one speaks English, or Chinese? and the same settings as I always puts me the Chinese version instead of English

ok, so far for what I have tested if this well and works perfectly the only thing the language as it is put as the level I always get the Chinese version, when I use the English language, should not put the 2 default if one speaks English, or Chinese? and the same settings as I always puts me the Chinese version instead of English.
or how to set the language? @CPJiNan

@Zarkness
You can try to change the "Language" item in the lang/zh_CN.yml file from zh_CN to en_US.

ok, so far for what I have tested if this well and works perfectly the only thing the language as it is put as the level I always get the Chinese version, when I use the English language, should not put the 2 default if one speaks English, or Chinese? and the same settings as I always puts me the Chinese version instead of English.
or how to set the language? @CPJiNan

@Zarkness You can try to change the "Language" item in the lang/zh_CN.yml file from zh_CN to en_US.

nothing, it would be easier to put in the settings.yml the language: <en_US> or <zh_CN>.


Will always show current week even when the pick up date has passed

I have pickup on Mondays and this prevents me from showing just one week. By the time next weeks info shows up it's already to late.

Fixed! When viewing only one week, it now shows the next week if the day has passed


Update sql-block.md

modify the document of the sql block rule
Proposed changes
Issue Number: close #xxx
Problem Summary:
Describe the overview of changes.
Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes/No)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

modify the document of the sql block rule


An 'Explore' home page similar to the desktop Main landing-page

What is the user problem or growth opportunity you want to see solved?
The Wikimedia Commons app needs a proper Explore home page that shows interesting contents for users to have some reason to actually use (and keep using) the app and discover contents.
It could be similar to the Discover page of the Wikipedia app and maybe use parts of its code. It could incorporate tiles of the desktop home page and make it possible to add or remove which tiles are shown (like for the Wikipedia app).
Which things are shown there could be discussed separately and should be customizable anyway – I never found desktop-WMC's "Media of the day" or "Picture of the day" interesting/useful or usually reasonable for example. One could show modular Tiles like these:

"Recently uploaded files featured in many Mediawiki pages", "Recent files added to your subscribed categories", "Highlighted content of your selected main-categories", "Highlighted categories", "Most viewed unseen media during the last 30 days", "Random quality images", "Recent audio files of music", "Highlighted audio file of music / video / … of your subscribed genre/category", "Media of Wikipedia articles in current month/year cat or current events cat", "Because you viewed many images of {this category}", "Images needing categories" and "Images needing captions" etc (suggested small edits if user is logged in), "Up-to-date charts and maps", "Random WMC campaign images", and so on (note: I expanded this list of ideas for tiles over time).

How do you know that this problem exists today? Why is this important?
It's important because of these benefits:

See the use-cases (more media that closes gaps, more contributors, more app users, retaining app users)
Making the app useful not just for uploading media; currently it's like an app to use for three minutes and never use again (if not for uploading) that is totally unengaging and not something one would expect from the well-resourced WMF or such a large website (and the largest structured free media repo) with tens–hundreds of thousands of app-users and a few times that potential users if the app was made to be truly useful/interesting
Improvements to this app are high-impact since most readers probably use WMC via smartphone by now and many of these use the app with many of the other mobile readers probably being open to switch to the app if they were prompted and the app was of high quality

I think this could be the most important issue in regards to WMC on mobile (with mobile rising in importance as more and more users are browsing or contributing via mobile phones).
Who will benefit from it?
The entire open media ecosystem and online global education such as Wikipedia, Wikimedia Commons as well as WMC app users and WMC users due to better and more contributions as well as more discoverable contents. Here are some use-cases and ways of how:

Discovering categories/subjects that miss media which they could then create/upload (I think a key goal should be to have more people upload useful media that is missing, many of these being illustrations, instead of only the 10,000th sunset photo)
Discovering interesting contents like Tilt–shift time-lapse videos or Microscopic images relating to biology
Finding their way around WMC by enjoying to use and extensively using the app

Anything else you would like to add?
No response

Hi @prototyperspective , thanks for your feedback! I'm curious if you have tried the "Explore" feature of the app yet, and if you have, how do you think it can be improved specifically? For instance, is it just the lack of customization that is the issue for you, or the way the UI is laid out, or the fact that it's not the "home" page?
Just to address this:

not something one would expect from the well-resourced WMF

AFAIK the WMF is not spending any resources on this app at the moment, and hasn't been since 2022. We're maintained solely by volunteers at this time. To the best of my knowledge, all avenues for WMF funding have been closed since the restructuring of the grants system.

Yes, I have used the Explore page of course which is why I created the issue – it merely shows a feed of random good-quality photos. I thought that page is already the default home page, at least for me it's the default page.
What can be improved specifically is what meant with "a proper Explore home page that shows interesting contents for users to have some reason to actually use (and keep using) the app and discover contents" in broad terms which I intended to specify with examples in the paragraph of "Recently uploaded files featured in many Mediawiki pages"….
Namely, diverse tiles for different types of contents just like the Wikipedia app. Please try out the Wikipedia app to see what I mean with tiles for different kinds of interesting contents. Likewise, the Main page of Wikimedia Commons desktop isn't a mere feed of semi-random good-quality images, it's a more useful landing page from where one can discover routes to all sorts of interesting contents such as by clicking on the category "Science".

AFAIK the WMF is not spending any resources on this app at the moment

It's a really horrible situation and I can't understand the WMF to any degree anymore with its millions of donated money but barely any funds for development of the piling backlog of bugs to fix and features+wishes to implement. Thanks for working on this as volunteers…even more reason for a dev-campaign-banner to get more volunteer developers involved displayed at the top of IT-related Wikipedia articles.

Below are the different sections I see on my Wikipedia app home activity. Please everyone post ideas for sections that would be interesting for the Commons app. Thanks!

Maybe it would easier/quicker to just show the Wikimedia Commons Main page until this is implemented which would be incredibly useful and is kind of expected. See the issue #5772 about that
Thanks for these screenshots, that's what I mean. However, it would be better if those images were smaller. I've edited the post to add all the ideas I had and I think the more modules/tiles people can enable there and personalize it to their liking, the more interesting the app would become and the more users there would be. Even the Wikipedia app shows a tile for "Picture of the day" but it's missing in the Commons app and just a page showing that would be good place to start implementing a feed that could then be added the the app (it doesn't have the be the landing page right away before there are more modules).
I think a Home Feed to explore is one of the three main key features – the other two being enabling uploads and  enabling browsing categories/searching. Just imagine if Twitter/X had no home feed, nobody would use it and I think the usefulness for WMC is pretty much comparable except for those who only use it to upload their smartphones photos.


StackExchange.Redis.StrongName in AspNetCore.SignalR.Redis is discontinued

The package Microsoft.AspNetCore.SignalR.Redis (<= 1.1.5) references StackExchange.Redis.StrongName (1.2.6). This package was discontinued by StackExchange about 9 month ago in favour of StackExchange.Redis, which is now strong named by default (https://github.com/Azure/aspnet-redis-providers/issues/107).
This reference causes problems when using other libraries, that already adopted the new StackExchange.Redis package. Other parts of AspNet Core seem to already have made transition to the new package. Any plans for the SignalR.Redis package?

Yes, we added a new package Microsoft.AspNetCore.SignalR.StackExchangeRedis in ASP.NET Core 2.2 (and the upcoming 3.0 release). We had to add a new package rather than upgrading the existing package because the StackExchange.Redis package changed package IDs, which is a breaking change. The older Microsoft.AspNetCore.SignalR.Redis package is discontinued in 3.0.
The APIs your app uses should be identical between the two, but be aware that if any other packages in your app reference the older StackExchange.Redis.StrongName package you could have both loaded into the app.
See https://github.com/aspnet/Announcements/issues/322 for details on this change.


Do you want to have the enhancement of existing game ? 😀 Describe yourself..
When we checkmate the AI, it alerts the following message:

even before we actually make the move.
Describe the solution you'd like
I will make it such that we take the move first and then it displays checkmate alert
Select program in which you are contributing
GSSoC24
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINE of this project.

Hey @VijaySamant4368 , You can't have another issue before completing the previous one 😀
you already have the following 1 open issues 👀 ! :
#3691

Hey @VijaySamant4368 ! Thank you so much for your raising the issue💗
It’s all yours, you can come anytime again and make some contributions! 🚀
Alone, we can do little, but together we can do so much! 😇

Hey @VijaySamant4368 !
Thank you for raising an issue 💗
You can self assign the issue by commenting /assign in comment 😀
Make sure you follow CODE OF CONDUCT and CONTRIBUTING GUIDELINES 🚀
Don’t Forget to ⭐ our GameZone🎮
Make sure you join our Discord🕹