HDDS-3090. Fix logging in OMFileRequest and OzoneManager

What changes were proposed in this pull request?
parameterized some logs introduced in HDDS-2940. Corrected log levels.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3090
Please replace this section with the link to the Apache JIRA)
How was this patch tested?
Executed Ozone FS interface unit tests. Only modified log messages.

/pending Review comments are not yet addressed.

/close


Adds ability to configure group display name and unique id LDAP attri…

Summary
Adds ability to configure LDAP attributes for group unique identifier and group display name.
TODO: Add description text.
Ticket Link
n/a
Checklist

[x] Ran make check-style to check for style errors (required for all pull requests)
[ ] Ran make test to ensure unit and component tests passed
[ ] Added or updated unit tests (required for all new features)
[ ] Needs to be implemented in mobile (link to PR or User Story)
[x] Has server changes (https://github.com/mattermost/mattermost-server/pull/9923)
[ ] Has redux changes (please link)
[x] Has UI changes
[x] Includes text changes and localization file (.../i18n/en.json) updates
[ ] Touches critical sections of the codebase (auth, posting, etc.)

@wiersgallak I have added the descriptions.


getting did you register the component correctly? error

Unknown custom element:  - did you register the component
correctly? For recursive components, make sure to provide the "name"
option.

After installing the package i imported that into my app.js:
timer.vue file content:
<template>
    <vue-countdown-timer
        @start_callback="startCallBack('event started')"
        @end_callback="endCallBack('event ended')"
        :start-time="'2018-10-10 00:00:00'"
        :end-time="1481450115"
        :interval="1000"
        :start-label="'Until start:'"
        :end-label="'Until end:'"
        label-position="begin"
        :end-text="'Event ended!'"
        :day-txt="'days'"
        :hour-txt="'hours'"
        :minutes-txt="'minutes'"
        :seconds-txt="'seconds'">
    </vue-countdown-timer>
</template>

app.js file content:
import Vue from 'vue'

import VueCountdownTimer from 'vuejs-countdown-timer';

Vue.use(Vuelidate)
Vue.use(VueCountdownTimer)

import Timer from './components/partials/timer.vue'

new Vue({
    store,
    components: {
        Timer,
        //
    }, computed: {}, mount: {}
}).$mount('#app')

and after that i try to use into html template as:
<login inline-template>

            ...

        <Timer></Timer>

</login>

my login.js content:
import {required, minLength, maxLength} from 'vuelidate/lib/validators'
import axios from "axios";
import {route} from "../../routes";

export default {
    data() {
        return {
            //
        }
    }
}

@MahdiPishguy importing the library into any Vue project and trying to use it exactly as in the readme.md does not work.
Why are you closing the issues reporting this?
You could simply delete the entire package, if is unusable.
For the end user it has pretty much the same effect.

@pishguy Hi, I got the same error. How did you resolve this?

@Firman95
sorry, unfortunately i have forgotten


Average response time/Throughput

Can average response time be calculated and displayed in the UI in order to sort operations of a particular service by slowest average response time?
Similarly is throughput calculable and displayable in the UI, or are we stepping into the realm of metrics here?

Ideally, the average response time panel would be a stacked graph where each layer of the stack represents a trace in the span a la New Relic (and probably other vendors).


Popup sideberry renders below browser content

Describe the bug
When hovering to open sideberry, I can only see the leftmost edge of it. I assume the rest is hidden behind the browser content.
Screenshots

Desktop:

OS: Windows 11 23H2
Firefox Version 133.0b1 Dev Edition

Add this at the end of userChrome.css
#sidebar-box {
  z-index: 4;
}


Remove nav walker and Bootstrap navbar

Cleaner walker moved to a Soil module: https://github.com/roots/soil/pull/73 (it no longer includes support for Bootstrap dropdowns)
Remove the Bootstrap navbar and use a more generic header template (remember, this is a starter theme (and we're becoming framework agnostic))

once upon a time, the starter theme had two header templates:

BS top nav
a simple header (like in this PR)

the simple header was removed in favor of just the BS top nav about a year ago
i never liked this change. personally, i have never used the BS top nav in a theme and it's one of the first things i remove when starting a new project.
we don't need to necessarily remove this right now [as the generator isn't yet ready], but i'd like to see it go. as you can see, we can remove 173 lines of code from the theme by doing this.
any feedback/thoughts? do you use the BS top nav?

I'm all in favor of cleaning up Sage to be more framework agnostic, but I personally use the provided nav walker as a starting point for many of my project's navigation (when I'm using Bootstrap). I think once the generator is in, it should be an addition if someone chooses Bootstrap (we should also provide the default still, IMHO).
For now, it might be a nice idea to keep it as a good starting point? Maybe add the "basic" navbar file back in?

I'm in favor of keeping it until the generator is there to replace it.
Not having to worry about the nav something I enjoy.

https://github.com/roots/soil/pull/73 is still compatible with bootstrap navigation menus because it adds the active class. it just doesn't support dropdowns.
if you added the BS topnav markup to the header template and had soil activated, you'd be bueno if you didn't need more than just the top level nav.

with this PR, if you just add nav-pills on the menu class you get:

with this PR, if you re-add the BS top nav markup, you get exactly what you're expecting (assuming soil is activated. if it's not, active menu item highlighting won't work out of the box):

@JulienMelissas

but I personally use the provided nav walker as a starting point for many of my project's navigation

what do you need from it - the dropdowns? (also, are you not using soil?)

@jdcauley

Not having to worry about the nav something I enjoy.

what does that even mean? are you using dropdowns or what?

Doesn't everyone use dropdowns? I think if you're building a theme, you have to take dropdowns into account if they're available on the backend. +1 on leaving this in until the generator comes.

Doesn't everyone use dropdowns?

erm, no.
and to re-iterate what i said earlier, this is a starter theme. there is plenty of stuff "available" on the backend that doesn't have any effect on the front-end.

I'm in favour of this cleaning up, but what about the BS collapsing navbar (and related button)?
This is maybe the first feature required from a mobile first theme.
Usually I override the default header with a custom one:
BS solution does not work in case of complex mobile header (menu button + search button + links) but in case of a simple header (logo + nav),  a BS nav is honey...
I'm thinking about _s starter theme: the only defualt javascript is the one handles toggling the navigation menu for small screens. :smile:
After the discussion listed under #1258 I tried to make Roots/Sage header compatible between different frameworks, using the same HTML structure...epic fail! :tired_face:
So maybe @jdcauley is right, until the generator is not ready.

we don't need to make decisions about how people are going to be doing their header navigation as far as collapsing and toggles go

Originally when this was posted, I was against it. I tend to use the Bootstrap navigation on most sites, as well as dropdowns. However, I think this opens up some interesting doors, and not even talking about the generator.
I don't think there's anything necessarily wrong with keeping this slim. Plus, if we start incorporating Composer for extra dependencies, like a Bootstrap Walker, this could really open the way for all other frameworks. It'd be pretty easy to convert that to a Foundation Walker and then include that much easier as well.
I know that a bit of the surrounding HTML will need to be added per project, unless the generator gets that sorted out. But for the time being, I think a gist will substitute nicely. It really won't take long to add it in if you're using Bootstrap or another framework.
So.... +1 for the PR

Ok, yea I'm using Soil, common Ben ;)
I think that as long as there are good resources for people who are used to the Bootstrap nav, I'm fine with this change. In fact, any movement towards a cleaner, leaner Sage (that still helps), I'm all for.

One of the things that I enjoy with Sage is that the navigation is ready to go and with rapid prototyping, not having to worry about setting up navigation is pretty beneficial.
I'm all for making Sage framework agnostic, so this makes total sense - but would love that there can be some way to easily be able to grab the walker to continue being able to quickly get a responsive navigation component working.

the collapsible navigation in the bootstrap navbar is one of many possible responsive menu patterns. i don't think we should force anyone to use a specific pattern off the bat.
the walker change in this PR wouldn't affect you using the bootstrap navbar markup with collapse button, all you'd need to do is update the header template to have the necessary markup.

I'm all for the change but only after either the generator is in place or we have created step-by-step documentation to get the nav up and running using 3rd party repos. It's not arduous to add the nav back in, but adding stuff is almost always more work than deleting stuff and walkers are something that I think a decent number of WordPress devs still struggle to get their head around.
If we do lose the code for the nav then I think we should seriously consider dropping Bootstrap entirely now, in favour of becoming framework agnostic with documentation explaining how to add your framework of choice.
I appreciate that this is going from one extreme to the other, but I don't see much point in half measures.

I agree with @Foxaii. There is nothing else left that is Bootstrap specific (that I can remember).

Also, there is no need to output <nav> if there is no primary navigation.

Also agree with @Foxaii here on removing Bootstrap if we decide to get rid of this, as long is there is some documentation on adding BS (and/or other frameworks) back in. Yay.

the nav walker in sage right now is plugin territory, imo. bootstrap or not.

this is getting merged. if you want the old header with dropdown support, please see this gist:
https://gist.github.com/retlehs/1b49f6c00d5140962d56
i encourage someone to create a fork of https://github.com/twittem/wp-bootstrap-navwalker and maintain it, since that repo seems like it needs some love. it would be cool if it had composer support, too.

if you want to use the 'clean walker', just make sure you have soil installed and activated on your site. bootstrap navs work fine with just the clean walker as long as you don't need dropdown support.

for anyone coming across this that wants the old functionality:
How to restore the Bootstrap nav [walkthrough] by @smutek

Hi,
I came across this pull request after extensively searching why the nav menu with bootstrap 3 & dropdown support was removed.
I think there's no such thing as "Bootstrap menu support without dropdown", as anyone familiar with Bootstrap knows there are dropdowns, and if WordPress supports them it's logical for them to think they should be available. At least I was very confused by this change and I'm no newbie at WP + Bootstrap development.
I suggest these work-around instructions are added to the main Sage website. I feel that it if advertises Bootstrap-based themes then it shouldn't be framework-agnostic. Or backwards: if it's going to be framework-agnostic, then it shouldn't advertise being based on Bootstrap.
These are my 2 cents as an avid PHP & WordPress developer. I use Bootstrap and like their dropdowns for simple projects (as, being simple, it's really a bad thing spending too much time on them for things that can be avoided").
Maybe a fully Bootstrap compatible fork of Sage is in order? Or perhaps just a section on the website explaining how to use all of its features.

by that logic, sage should support every BS component out of the box.
sage is a starter theme and is definitely bootstrap based out of the box. we shouldn't need to update our website to mention that the nav support has changed.
this theme goes through changes all of the time, and if you're using it you should be keeping track of these changes.
your "extensive search" could have been just looking at the changelog which would have brought you here.
i understand where you are coming from and many others also share your opinion. that said, these decisions are made for a reason and if we tried to please everyone, sage would probably be a big pile of shit.
Sent from my iPhone

On Jun 10, 2015, at 1:29 PM, Leonardo Quijano<EMAIL_ADDRESS>wrote:
Hi,
I came across this pull request after extensively searching why the nav menu with bootstrap 3 & dropdown support was removed.
I think there's no such thing as "Bootstrap menu support without dropdown", as anyone familiar with Bootstrap knows there are dropdowns, and if WordPress supports them it's logical for them to think they should be available. At least I was very confused by this change and I'm no newbie at WP + Bootstrap development.
I suggest these work-around instructions are added to the main Sage website. I feel that it if advertises Bootstrap-based themes then it shouldn't be framework-agnostic. Or backwards: if it's going to be framework-agnostic, then it shouldn't advertise being based on Bootstrap.
These are my 2 cents as an avid PHP & WordPress developer. I use Bootstrap and like their dropdowns for simple projects (as, being simple, it's really a bad thing spending too much time on them for things that can be avoided").
Maybe a fully Bootstrap compatible fork of Sage is in order? Or perhaps just a section on the website explaining how to use all of its features.
—
Reply to this email directly or view it on GitHub.

That makes no sense. Forcing developers to comb through changelogs when they can see a formal documentation section that explains what happens and why is just bad business.
Every respected framework or library out there makes an effort to keep developers updated on changes, to guide people who used to use a certain functionality on how to enable it again (cleary and without looking at logs), and to support basic things like a bootstrap dropdown.
I'm offering suggestions based on what I know from years of experience in the field. These kinds of obscure decisions only drives people away, and your answer just points out clearly that you didn't get it: if I had to spend 2 hours figuring out why the new version of sage broke old functionality, then there's something really wrong with the development process.
Anyway, I'll use the starter theme if it makes sense, or migrate somewhere else if it doesn't. Though I was hoping a more open minded and professional approach, though.

are you trying to say that the changelog isn't the right place to check for changes? because you're wrong.
we have "years of experience" with running successful open source projects & your attitude/sense of entitlement is nothing new to us.
this isn't the right place to voice your complaints. you can use the roots discourse for that, as stated in our contributing guidelines.
Sent from my iPhone

On Jun 11, 2015, at 1:51 PM, Leonardo Quijano<EMAIL_ADDRESS>wrote:
That makes no sense. Forcing developers to comb through changelogs when they can see a formal documentation section that explains what happens and why is just bad business.
Every respected framework or library out there makes an effort to keep developers updated on changes, to guide people who used to use a certain functionality on how to enable it again (cleary and without looking at logs), and to support basic things like a bootstrap dropdown.
I'm offering suggestions based on what I know from years of experience in the field. These kinds of obscure decisions only drives people away, and your answer just points out clearly that you didn't get it: if I had to spend 2 hours figuring out why the new version of sage broke old functionality, then there's something really wrong with the development process.
Anyway, I'll use the starter theme if it makes sense, or migrate somewhere else if it doesn't. Though I was hoping a more open minded and professional approach, though.
—
Reply to this email directly or view it on GitHub.

@leoquijano, while I value your opinion I disagree with a few points you've made above.
First of all, a changelog is just that: a place to log changes. We do make an effort to keep people involved and up to date using discourse, blog posts, and github issues, and we've also been very responsive to people on discourse asking where it went (https://discourse.roots.io/t/what-does-the-a-default-sage-install-look-like-screenshot/3975/2 - there are many others). In your time you spend searching you probably could have taken 2 minutes on our discourse making a new post or doing a search, I would have seen your post because I get notifications, and in my free time I would have quickly responded with a link.
Second, not every popular framework has beautiful changelogs like you're requesting. Let's take Bootstrap itself for example since you're experienced with using it - remember when they removed subnav menu support? Bootstrap doesn't have that in their documentation, but it's documented in a Pull Request just like we do. It's not even in their changelog for version 3.0.0. That made people upset. Boohoo, people still use Bootstrap, and tons of posts on the internet went up for how to fix it. Problem solved.
Third, most projects have growing pains. We can't always be 100% what we want to be. Right now, we're trying to move Sage towards being framework agnostic but we also understand many people still love and use Bootstrap and until we have a better solution, at least for now, we're keeping some of it's functionality. That's the decision of, what Ben said, is a pretty experienced team. Look at all of our green squares!
Fourth, while we love to hear people are using Sage to build awesome websites, you don't have to use it. Fork it, add your full Bootstrap support back and BOOM you've got your own starter theme, or use something else.

@retlehs Ben, Thanks for developing such a great starter theme on the bleeding edge. I whole hardily agree with you. I just started a new project with Sage 3 days ago and when I saw the navigation I thouth things were busted. I search the discourse and did not find anything so I asked if anyone had a screen shot of the home page and was imediatly directed to a new install of the theme. I was also notified about the way to restore the navwalker. The response was within 30 minutes. Great project and don't waste time on these issues. Cheers!

Hi Julien. Thanks for your answer. That's the kind of thoughtful argument I was looking for.
This was not a support request. That's the reason I didn't use the forums (I don't use them since I'm used to "RTFM" as old hackers used to say). Also, keep in mind that some people are used to combing through changelogs. Some aren't.
It's a suggestion: put a Bootstrap section in the main website with instructions of how to work with common Bootstrap things. Maybe you will end up with even a module or subproject.
It's probably not something that takes too much time. I would gladly offer to do it if I thought it would be well received.

That's the kind of thoughtful argument I was looking for.
Don't get on your computer looking for an argument, that took 15 minutes out of my day.

On Jun 11, 2015, at 3:25 PM, Leonardo Quijano<EMAIL_ADDRESS>wrote:

Hi Julien. Thanks for your answer. That's the kind of thoughtful argument I was looking for.
This was not a support request. That's the reason I didn't use the forums (I don't use them since I'm used to "RTFM" as old hackers used to say). Also, keep in mind that some people are used to combing through changelogs. Some aren't.
It's a suggestion: put a Bootstrap section in the main website with instructions of how to work with common Bootstrap things. Maybe you will end up with even a module or subproject.
It's probably not something that takes too much time. I would gladly offer to do it if I thought it would be well received.

I would welcome it and started a discussion on discourse which I’d love for you to help move this thread to: https://discourse.roots.io/t/abstracting-away-from-bootstrap/4028

—
Reply to this email directly or view it on GitHub.

It's a suggestion: put a Bootstrap section in the main website with instructions of how to work with common Bootstrap things. Maybe you will end up with even a module or subproject.

Why would we add a section to the main website with instructions for how to work with Bootstrap when we've already made the decision to move away from Bootstrap?
You're essentially demanding that we provide documentation for something that has nothing to do with Sage, per se. A Bootstrap nav walker could be used for any WordPress theme that loads Bootstrap, If you're choosing to use Bootstrap, feel free to find a navbar implementation that works for you. It's patently absurd to tell us that it's our duty to document how integrate your favorite Bootstrap component into a WordPress theme.

Why would we add a section to the main website with instructions for how to work with Bootstrap when we've already made the decision to move away from Bootstrap?

Well, because that's what your website says, that it's Bootstrap based:
"Sage is a WordPress starter theme based on HTML5 Boilerplate, gulp, Bower, and Bootstrap, that will help you make better themes."


Google Authenticator. "Error: Invalid token provided"

I don't seem to be able to get Google authenticator to work with this tool. I register a device and then try and login and it states that the token is invalid, even though it is valid.
Do you have any idea how to get this working, as I urgently need to derive AWS keys for authenticating.
kas@K:~$ gimme-aws-creds --profile esure --username<EMAIL_ADDRESS>--register_device
Please enter password for encrypted keyring:
Using password from keyring for<EMAIL_ADDRESS>Multi-factor Authentication required.
token:software:totp( GOOGLE<EMAIL_ADDRESS>selected
Enter verification code: 906171
Device token saved!
kas@K:~$ gimme-aws-creds --profile esure --username<EMAIL_ADDRESS>Please enter password for encrypted keyring:
Using password from keyring for<EMAIL_ADDRESS>Multi-factor Authentication required.
token:software:totp( GOOGLE<EMAIL_ADDRESS>Enter verification code: 040174
Error: Invalid token provided

I'm having a hard time thinking of a way that gimme-aws-creds could be messing this one up, particularly since I know of a couple of users that have been using Google Authenticator with this.  Do we know for sure this isn't an MFA configuration issue in Okta?

I am facing the same issue.
(gimme-aws-creds) abhishek-mba:gimme-aws-creds abhishek$ gimme-aws-creds
Using password from keyring for abhishek
Password for abhishek:
Do you want to save this password in the keyring? (y/n) n
Multi-factor Authentication required.
Pick a factor:
[0] token:software:totp( GOOGLE<EMAIL_ADDRESS>Selection: 0
Enter verification code: 605889
Error: Invalid token provided

Does gimme-aws-creds support Google Authenticator?

I get this same error on all Authentication methods using my Mac.
It works with the same token factor option on a centos 7 vm - anyone have this working on mac?

RE: my specific issue was a config error
my config had
gimme_creds_server = app_url

where it needed to be
gimme_creds_server = appurl

Closing for now - I think this is an issue that has been addressed.
If found again, we can re-open.


build python3 w/ sqlite extension loading support

fixes #488

Is there some reason this can't be merged?

Support for this was added in #1246 for Python 3.10 and above :-)


Is the integral of exp(-1/x) is divergent over the [-1,1]?

@AhmedCode99 Yes, the divergence is likely due to the behavior of the function near x=0, where it becomes undefined or extremely large.

yes


clearsigning

Hi again,
Is it possible to clear sign with this library?
I've tried a few variations (can share code) but I can't get a signature which verifies.
Thanks
Pat

There's no clearsigning helper just now, but all the machinery is there.  Should just be a matter of sticking the material you want to clearsign in a type-u data packet, signing that message, extracting the signature packet and ASCII-armoring that.  I think.

Thanks, I don't get much of that (PGP is another world for me), but I get everything bar the type-u data packet and I am sure I can have a try at that
Thanks again

If I get it working I will send a sample back

Hey
So here is my code, does this seem "clearsigned" to you?
It doesn't work for me (note i am using mudhairless's PSR-2 version of your code)
$options = array();
$options['format'] = "u";
$typeUPacket = new \OpenPGP\Packets\LiteralDataPacket($hashedDocument, $options);
$keyASCII = file_get_contents($this->getKeyPath());
$util = new \OpenPGP\Util();
$unarmor = $util->unarmor($keyASCII, 'PGP PRIVATE KEY BLOCK');
$keyEncrypted = \OpenPGP\Message::parse($unarmor);
foreach ($keyEncrypted as $p) {
$key = \OpenPGP\Crypt\Symmetric::decryptSecretKey($this->getPassPhrase(), $p);
$rsa = new \OpenPGP\Crypt\RSA($key);
$m = $rsa->sign($typeUPacket);
}
$util = new \OpenPGP\Util();
$headers = array(
"Version" => "GnuPG v2"
);
$message = $util->enarmor($m->packets[0]->data[0], "PGP SIGNATURE", $headers);
$message = "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n" . $hashedDocument . "\n" . $message;
echo $message;

If it helps it seems like there are extra lines

Here are two examples
The first is via PHP, second via command line
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
8f3c7c6932f7e8b9f8773152212c5f65f5c07ca3
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
gGTHzEx8Ye5bMb5s7ih7KQN+T5Xx/cwrXgwHLsyfTBKP7zWlCFHel9avhxZM4Ojk
hXuilOTKNsebtNUesFZBwB6PxTrVw0A8sufh5rfffPELI+goyD2Hrck0umHFhUaE
MyYiHpZfplNj2hBotqHFc4FOJXQh1SOtwNibyaF3i9hejqbjPxfMgisEZqQmYGGf
TA4bJoe1mOQzj8MyHusbYFSX0wBlcpx06lCMuPxEaPxWpoqH4jKNbnroqnE3KPG2
q+c3eRRtDuUbcWyQx7Z+EML49oWpmh1NzMTrLJbCVL0+QawA0MTvW7ZsMrL9Y7u2
AQAqo4T+e1wLSyFLMeVh1A==
=SIUU
-----END PGP SIGNATURE-----
And if I use command line --clearsign
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
8f3c7c6932f7e8b9f8773152212c5f65f5c07ca3
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQEcBAEBCAAGBQJVgYPjAAoJEO1RiXtdOTQim9MH/jhSkko67FVuT3OJ0suxzT3Z
gIDSDg6wN046C2OAqn3zcCH7htKs8bhR8NyxQ4UdeuweFT85CzyqTLt8RgiHACqw
aC4i91s9np87Pqy2qn+Ba16p+5u4QCtIxfkWzcPMnK41nakgqsgTh+yOASoCwaUm
eB7uI1wRKSVjZkskIkODK956mr4A5xLEC2/7b4z6p+KKBrayrSxtkd1M9L690VNZ
sKHvr6tHp598+i3h7yCokMoQd6+TYkoIc3KlE1Cddlf4RgKCoLwknNfdjz3c0HnB
g8f5JOQhBodZEa3LDk3BcVO/rA8npGCGMSjkHnGoysX2GLqV+nuZ8ZKu9r+S2Ao=
=SJY0
-----END PGP SIGNATURE-----

I have added a working example at https://github.com/singpolyma/openpgp-php/blob/master/examples/clearsign.php  --  I think the problem with your code is that you are not calling to_bytes

Thanks so much for the help
I now have a merge of the two examples
require_once dirname(FILE).'/lib/openpgp.php';
require_once dirname(FILE).'/lib/openpgp_crypt_rsa.php';
require_once dirname(FILE).'/lib/openpgp_crypt_symmetric.php';
//filepath is just a string pointing to a password free key
$keyASCII = file_get_contents($filepath);
$keyEncrypted = OpenPGP_Message::parse(OpenPGP::unarmor($keyASCII, 'PGP PRIVATE KEY BLOCK'));
foreach($keyEncrypted as $p) {
if(!($p instanceof OpenPGP_SecretKeyPacket)) continue;
$key = OpenPGP_Crypt_Symmetric::decryptSecretKey($passphrase, $p);
}
/* Create a new literal data packet /
$data = new OpenPGP_LiteralDataPacket($message, array('format' => 'u', 'filename' => 'stuff.txt'));
/ Create a signer from the key /
$sign = new OpenPGP_Crypt_RSA($key);
/ The message is the signed data packet /
$m = $sign->sign($data);
/ Generate clearsigned data */
$packets = $m->signatures()[0];
echo "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n";
echo preg_replace("/^-/", "- -",  $packets[0]->data)."\n";
echo OpenPGP::enarmor($packets[1][0]->to_bytes(), "PGP SIGNATURE");
Doesn't produce valid sigs. I must be missing something obvious, but not sure what. I wonder if it is how the key is opened?

When I do gpg --verify it says the key is not cross certified (if that helps, this is all a bit over my head)

@pgogy that sounds like a problem with the key you're signing with.

Yeah, that's my conclusion. For ref
I do
gpg --gen-key
RSA and DSA
2048
So not sure if the issue is here

Try with an RSA/RSA key maybe?

Yep, a sign only key worked.
Thanks for your time, no idea what the difference was (the PSR-4 class didn't work either)
If you're on gittip or some such just let me know
Thanks a lot
Pat


Merge solo and master

Feature Request
Detailed Description

The two repositories have as of yet not diverged much.  Is there any downside to merge solo into master and delete the solo branch?  We could cut a v0.1.x branch to preserve master as-is, but I'm not sure we need to yet.  The downside is that we may break compatibility with current (master) dispatch in the near future as we add features and align with Knative.  However, right now, we should be pointing folks towards dispatch-solo as it's a good way to get started.
The second question is if we should remove k8s support from solo then?  Or if that support should change?  This is debatable as it largely depends on the maturity of Knative and the dispatch-Knative branch.
Context

Possible Implementation

Complexity

[ ] Low - Simple enhancement or bug fix, no architectural changes or refactoring
[ ] Medium - Change requires some thought, but is relatively isolated
[ ] High - Significant architectural change or large refactor

My thoughts:

If we are decided on going solo + knative, my understanding is that we will have the same APIs, just different underpinnings, reduced feature set, and different installation method. if so, we should strive to merge everything into single branch, and go from there. This means more upfront work but will reduce the pain later.
I'm all in on making the hard decision here and removing the k8s support. It's tempting to keep it, but IMHO focusing on one (well, two things) is more important.

Typically master is used for the core development which for dispatch longer term is likely the knative branch. Per what @kars7e mentioned, if there will be commonality is solo and knative then do that in master. Otherwise, switching master to solo in the interim doesn't make as much sense to me. Oh, and keep in mind needing to merge the knative branch at some point.

OK, so I'm hearing a few things:

merge the knative branch to master, accept that it's not full featured yet.
remove k8s support from solo (k8s support will be from knative)

So, should we then just keep solo as a branch (sounds like yes)?

One more question, should we move the CLI to a dispatchframework repo?

Sounds like a good idea. API / client library as well.

For me, merging everything to master (including solo). I'm not sure how separate branch for solo helps here. I'm neutral on whether CLI should be a separate repo.

We could also move the client out, though I'd probably hold off on that.  I think we should have an in-person/zoom chat about whether to combine the servers into a single branch, though I can say from experience, switching branches right now is painful.

I like the idea to use knative branch as master.
But for solo if we don't plan to merge it back to future master (read knative) then we could as well move it to a new repo dispatch-solo. Any shared code can be later moved to another repo say dispatch/pkg. CLI / Client could follow the suite.

I can create a new repo if that makes sense (I think it does).  I have to go through OSSTP anyway for solo.  Do we need to chat more, or is this a plan?

I also like the idea of knative as master and removing k8s from solo. My only concern is making sure that changes to the cli don't break e2e tests in the knative or solo repos.


[Feature Request] Remove dependancy on firebase_database ^10.0.0 / Add support to work with firebase_database ^11.0.0

Hello!
Is there a reason that this package does not support firebase_database: ^11.0.0?

Is there a reason that this package does not support firebase_database: ^11.0.0?

Yes, because it was published only 2 days ago.

Is there a reason that this package does not support firebase_database: ^11.0.0?

Yes, because it was published only 2 days ago.

Sorry about that I did not notice that! Thank you so much for the reply!!

Will update all firebase dependencies in an upcoming PR.


Make it easier to test functions that use Plan Datatypes with rspec-puppet

Use Case
The Plan Datatypes live in the boltlib module that ships with the bolt gem which makes it hard to test functions that rely on them.
Describe the Solution You Would Like
tbd
Describe Alternatives You've Considered

A BoltSpec function that returns the path to bolt-modules inside the gem.
CI that publishes bolt-modules to forge
A helper for testing functions in a plan like context(ie with inventory and executor configured)

Additional Context
Add any other context or screenshots about the feature request here.

For the time being I ended up mocking the TargetSpec type via let(:pre_condition) { "type TargetSpec = Variant[String[1], Target, Array[TargetSpec]]" } in the function spec file.
Or if using the debugger I can define the type via type TargetSpec = Variant[String[1], Target, Array[TargetSpec]]
Additionally, I also added these symlinks in my fixtures.  I don't know if any of these options are the best route but they do seem to solve some issues and create new ones too.
symlinks:
    "peadm": "#{source_dir}"
    out: "#{source_dir}/.bundle/gems/bolt-1.49.0/bolt-modules/out"
    ctrl: "#{source_dir}/.bundle/gems/bolt-1.49.0/bolt-modules/ctrl"
    file: "#{source_dir}/.bundle/gems/bolt-1.49.0/bolt-modules/file"
    system: "#{source_dir}/.bundle/gems/bolt-1.49.0/bolt-modules/system"
    boltlib: "#{source_dir}/.bundle/gems/bolt-1.49.0/bolt-modules/boltlib"

This should also work which is similar to above but dynamic and will work on every machine.
This would need to be added to PDK templates or possible as a default in the puppet-spec_helper gem.
# spec_helper.rb or spec_helper_local.rb
spec = Gem::Specification.latest_specs.find do |spec|
  File.exist?(File.join(spec.full_gem_path, File.join('lib', 'bolt')))
end
RSpec.configure do |c|
  c.basemodulepath = File.join(spec.full_gem_path, 'bolt-modules')
end


SPL BUG -  Detect Rare Executables

From LinkedIN: Antony Bowesman
I'm a Splunk Trust member, working with Splunk Security @Deloitte - I've got some feedback on one of your anomaly detection searches, which is fundamentally flawed... (tstats count + rare will never work)
https://research.splunk.com/endpoint/44fddcb2-8d3b-454c-874e-7c6de5a4f7ac/
I'm looking at some of the ESCU searches for a client and noticed they have implemented the Detect Rare Executables search. As it was generating Notable events, I checked it and found that the search is flawed, both technically and conceptually.
Conceptually the time windows of the two tstats statements is the same, which seems a little odd in that the second tstats is effectively defining 'rare' as those in the current search time window, rather than determining rarity over a longer time window.
Technically, with tstats count by Processes.process_name, that will generate a single count for each process name, so when you do rare on process name, there is only ONE process name, so you will effectively get the first 30 (alphabetically ordered) items from that count NOT the ones with the lowest tstats count!
A correct search would follow tstats count with sort count | head 30.
Conceptually, the filtering of executables inside the subsearch is done AFTER the 30 'rare' items, so if you have 60 items after tstats that would be considered 'rare', then you cannot filter out the allowed ones before you determine rarity, so in practice, you may filter out all 30 left after the rare command and never find the true rare ones.
The final filter seems rather pointless in that you have already exercised two filters, but there may be other filtering requirements, so that's ok.
Given that this search is both part of ESCU and Splunk Security Essentials, it would be good to fix the technical deficiencies, as Splunk customers learn search techniques from these searches.
I hope I have made myself clear enough - if there is a bug route to file bugs in these searches, please let me know.
Cheers
Antony

The technically broken SPL is in the subsearch, it currently does
| tstats count from datamodel=Endpoint.Processes by Processes.process_name
| rare Processes.process_name limit=30

but the tstats will produce ONE row per process name, so rare process name is impossible to determine, as frequency is 1 per name. The search could do this instead
| tstats count from datamodel=Endpoint.Processes by Processes.process_name
| sort count
| head 30

As for conceptual flaws, an initial filter should be possible after the first tstats
| tstats `security_content_summariesonly` count values(Processes.dest) as dest values(Processes.user) as user min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.process_name
| `detect_rare_executables_filter`

so that any rarity calculation done later only works on potential bad candidates.

As for the general concept of 'rarity', if you are searching an X minute window of executables and then use that same list as a 'rare' filter in the subsearch, you are likely to produce a higher proportion of false positives as 'rarity' will occur more frequently, especially outside high usage hours.
It seems like the only purpose of the subsearch filter is to prevent the 'loss' of the dest, user, first/last times that would occur with rare. As rare is flawed, the entire search could be
 tstats `security_content_summariesonly` count values(Processes.dest) as dest values(Processes.user) as user min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.process_name
| `filter_rare_process_allow_list`
| sort count
| head 30
| rename Processes.process_name as process
| rex field=user "(?<user_domain>.*)\\\\(?<user_name>.*)"
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `detect_rare_executables_filter`

So, the search now does

Find all executables in window
Remove all allowed processes
Take least executed 30 results
Perform post data fixups
Final filter

This still does not address the concept of 'rarity' within a longer term window, but at least will be functional.

Hello @bowesmana : thanks for all these details, after testing this a bit on some live data: the current search logic is definitely flawed and needs a re-haul. I tested your proposed search and it performs better! We will update this in our upcoming releases
Like you mentioned, this updated SPL doesnt address the concept of rarity which is something I want to dive into and see if we can solve it the right way. Thank you again!

updated PR : https://github.com/splunk/security_content/pull/2449


Add an option to djv_convert to toggle pre-multiplied alpha

Original ticket:
https://sourceforge.net/p/djv/feature-requests/78/

I've decided to limit the scope of DJV and not include export functionality.


Inference results don't match test results

Checklist

I have searched related issues but cannot get the expected help.
I have read related documents and don't know what to do.

Describe the question you meet
Hello, I trained Resnet-101 on my custom dataset. Then I tested the model with the official test tool on my test set and got an accuracy of 91.44%. However, when I try to inference the test set image one by one, I found the model can't classify images correctly.   My data set has 3 labels '3h','5h' and '8h', The model will classify everything into ''8h'. But when I reload the model, it might predict everything into '5h'.
To control, I tried different models on different dataset and this issue still existed. Which result should I believe?
Thank you!
Post related information

The output of pip list | grep "mmcv\|mmcls\|^torch"

mmcls                         0.18.0
mmcv-full                     1.4.0
torch                         1.8.1
torchaudio                    0.8.0a0+e4e171a
torchvision                   0.9.1

Your config file if you modified it or created a new one.

model = dict(
    type='ImageClassifier',
    backbone=dict(
        type='ResNet',
        depth=101,
        num_stages=4,
        out_indices=(3, ),
        style='pytorch'),
    neck=dict(type='GlobalAveragePooling'),
    head=dict(
        type='LinearClsHead',
        num_classes=3,
        in_channels=2048,
        loss=dict(type='CrossEntropyLoss', loss_weight=1.0),
        topk=(1, 5)))
dataset_type = 'ImageNet'
img_norm_cfg = dict(
    mean=[420.57, 683.84, 1104.41], std=[100.3, 50.3, 150.6], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile', color_type='unchanged', to_float32=True),
    dict(type='RandomResizedCrop', size=224),
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
    dict(
        type='Normalize',
        mean=[420.57, 683.84, 1104.41],
        std=[100.3, 50.3, 150.6],
        to_rgb=True),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']),
    dict(type='Collect', keys=['img', 'gt_label'])
]
test_pipeline = [
    dict(type='LoadImageFromFile', color_type='unchanged', to_float32=True),
    dict(type='Resize', size=(256, -1)),
    dict(type='CenterCrop', crop_size=224),
    dict(
        type='Normalize',
        mean=[420.57, 683.84, 1104.41],
        std=[100.3, 50.3, 150.6],
        to_rgb=True),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]
data = dict(
    samples_per_gpu=32,
    workers_per_gpu=4,
    train=dict(
        type='ImageNet',
        data_prefix='/mnt/data/processed/microglia/ImageNet_merge/train',
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='unchanged',
                to_float32=True),
            dict(type='RandomResizedCrop', size=224),
            dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'),
            dict(
                type='Normalize',
                mean=[420.57, 683.84, 1104.41],
                std=[100.3, 50.3, 150.6],
                to_rgb=True),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='ToTensor', keys=['gt_label']),
            dict(type='Collect', keys=['img', 'gt_label'])
        ],
        classes='/mnt/data/processed/microglia/ImageNet_merge/classes.txt'),
    val=dict(
        type='ImageNet',
        data_prefix='/mnt/data/processed/microglia/ImageNet_merge/test',
        ann_file='/mnt/data/processed/microglia/ImageNet_merge/test/test.txt',
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='unchanged',
                to_float32=True),
            dict(type='Resize', size=(256, -1)),
            dict(type='CenterCrop', crop_size=224),
            dict(
                type='Normalize',
                mean=[420.57, 683.84, 1104.41],
                std=[100.3, 50.3, 150.6],
                to_rgb=True),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ],
        classes='/mnt/data/processed/microglia/ImageNet_merge/classes.txt'),
    test=dict(
        type='ImageNet',
        data_prefix='/mnt/data/processed/microglia/ImageNet_merge/test',
        ann_file='/mnt/data/processed/microglia/ImageNet_merge/test/test.txt',
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='unchanged',
                to_float32=True),
            dict(type='Resize', size=(256, -1)),
            dict(type='CenterCrop', crop_size=224),
            dict(
                type='Normalize',
                mean=[420.57, 683.84, 1104.41],
                std=[100.3, 50.3, 150.6],
                to_rgb=True),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ],
        classes='/mnt/data/processed/microglia/ImageNet_merge/classes.txt'))
evaluation = dict(interval=10, metric='accuracy', metric_options=dict(topk=1))
optimizer = dict(type='Adam', lr=0.0001, weight_decay=0.001, amsgrad=False)
optimizer_config = dict(grad_clip=None)
lr_config = dict(policy='step', step=[20, 50, 70], gamma=0.1)
runner = dict(type='EpochBasedRunner', max_epochs=100)
checkpoint_config = dict(interval=10)
log_config = dict(interval=100, hooks=[dict(type='TextLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
work_dir = 'merge_resnet_100e_128'
gpu_ids = range(0, 1)

you can use more args in test.py to find where is the problem, for example,--out, --out-items, --show-dir. for more details please refer to the code
Maybe, your gt_label is wrong when training,you can use tool vis_pipeline to check, refer to doc.

Hello, as I have already reproduced the problem on different data set (in my case, different cell types data), it might not be the gt_label problem. I will follow your advice and try to find out the reason.
Thank you!

The model will classify everything into ''8h'. But when I reload the model, it might predict everything into '5h'.
It looks like you failed to load the checkpoint, how do you load it?

Hello, I use the command below:
config_file='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/resnet_config.py'
checkpoint_file ='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/epoch_100.pth'
device = 'cuda'
model = init_model(config_file, checkpoint_file, device=device)
And the output is:
/home/gabriel/anaconda3/envs/mmlab/lib/python3.8/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above.
and should_run_async(code)
load checkpoint from local path: /home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/epoch_100.pth

So how do you "inference the test set image one by one"?

result={'8h':0,'5h':0,'3h':0}
for img in img_list:
image_seg = TIFF.open(img,'r').read_image().astype(np.int16)
res = inference_model(model, image_seg)
result[res['pred_class']] +=1
I also tried to check individual inference on notebook, and one of the result is:
{'pred_label': 2, 'pred_score': 1.0, 'pred_class': '8h'}.

Looks like no problem, can you provide your dataset class?

It's simply 3h, 5h and 8h. I just tried to use tools/test.py, the accuracy is 91.44%, the recall is 91.51%.
But If I inference the test/5h/*.tif, the result would be {'8h': 10620, '5h': 8, '3h': 2}

It's simply 3h, 5h and 8h. I just tried to use tools/test.py, the accuracy is 91.44%, the recall is 91.51%. But If I inference the test/5h/*.tif, the result would be {'8h': 10620, '5h': 8, '3h': 2}

Can you provide your dataset class?

It's simply 3h, 5h and 8h. I just tried to use tools/test.py, the accuracy is 91.44%, the recall is 91.51%. But If I inference the test/5h/*.tif, the result would be {'8h': 10620, '5h': 8, '3h': 2}

Can you provide your dataset class?

Sorry I don't get it, you mean the dataset config file or something?

Well, I get it,
Can you try it with:
import mmcv
from mmcls.apis import inference_model, init_model, show_result_pyplot
import glob,random
import glob, os
from libtiff import TIFF,TIFFimage
import matplotlib.pyplot as plt
import numpy as np
import random

config_file='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/resnet_config.py'
checkpoint_file ='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/epoch_100.pth'
device = 'cuda'
model = init_model(config_file, checkpoint_file, device=device)

img_list=glob.glob('/mnt/data/processed/microglia/ImageNet_merge/test/5h/*.tif')

result={'8h':0,'5h':0,'3h':0}
for img in img_list:
    res = inference_model(model, img)
    result[res['pred_class']] +=1

print(result)
print(res)

Well, I get it, Can you try again with:
import mmcv
from mmcls.apis import inference_model, init_model, show_result_pyplot
import glob,random
import glob, os
from libtiff import TIFF,TIFFimage
import matplotlib.pyplot as plt
import numpy as np
import random

config_file='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/resnet_config.py'
checkpoint_file ='/home/gabriel/algorithm/tasks/microglia/classification/merge_resnet_100e_128/epoch_100.pth'
device = 'cuda'
model = init_model(config_file, checkpoint_file, device=device)

img_list=glob.glob('/mnt/data/processed/microglia/ImageNet_merge/test/5h/*.tif')

result={'8h':0,'5h':0,'3h':0}
for img in img_list:
    res = inference_model(model, img)
    result[res['pred_class']] +=1

print(result)
print(res)

Surprisingly it works! It seems to be some data type problem, and I will follow your pipeline in the future. I really appreciate your help!
BTW, could you please tell me why it works with a file path rather than an array?

It's simple, here is the original code in mmcv to read the image
        if backend == 'turbojpeg':
            with open(img_or_path, 'rb') as in_file:
                img = jpeg.decode(in_file.read(),
                                  _jpegflag(flag, channel_order))
                if img.shape[-1] == 1:
                    img = img[:, :, 0]
            return img
        elif backend == 'pillow':
            img = Image.open(img_or_path)
            img = _pillow2array(img, flag, channel_order)
            return img
        elif backend == 'tifffile':
            img = tifffile.imread(img_or_path)
            return img
        else:
            flag = imread_flags[flag] if is_str(flag) else flag
            img = cv2.imread(img_or_path, flag)
            if flag == IMREAD_COLOR and channel_order == 'rgb':
                cv2.cvtColor(img, cv2.COLOR_BGR2RGB, img)
            return img

Since you didn't specify the backend, mmcv use the cv2 as the backend to load images in the LoadImageFromFile pipeline.
I think the outputs of different loading backends are different.
img_path = 'tiff_img_file'
img1 = TIFF.open(img_path,'r').read_image().astype(np.int16)
img2 = cv2.imread(img_path, 'unchanged').astype(np.float32)

BTW, if you directly pass the file path to inference_model, it will use LoadImageFromFile in your pipeline to load the image

I got it, thanks a lot!!


about point cloud topics

Hi,
I can not find point cloud topics when I rosrun kinect2_bridge. I am searching for these:
/kinect2/depth_lowres/points
/kinect2/depth_highres/points
Did I miss something?
referance: https://github.com/code-iai/iai_kinect2/tree/master/kinect2_bridge#point-cloud-topics
I need cloud data so that I can process image and extract objects and peoples and so on.
What we have right now is depth data right? If so, how can I obtain pointcloud data from them. I mean what do those numbers stand for when I get using /kinect2/depth/image?
They look like
....... , 2, 33, 2, 33, 2, 32, 2, 32, 2, 31, 2, 32, 2, 31, 2, 30, 2, 30, 2, 30, 2, 30, 2, 31, 2, 0, 0, 29, 2, 0, 0, 0, 0, 28, 2, 28, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 35, 2, 34, 2, 36, 2, 36, 2, 38, 2, 38, 2, 0, 0, 39, 2, 40, 2, 39, 2, 39, 2, 39, 2, 40, 2, 40, 2, 42, 2, 41, 2, 40, 2, 0, 0, 40, 2, 40, 2, 39, 2, 40, 2, 40, 2, 41, 2, 41, 2, 40, 2, 40, 2, 40, 2, 41, 2, 42, 2, 42, 2, 42, 2, 44, 2, 44, 2, 44, 2, 42, 2, 43, 2, 42, 2, 43, 2, 42, 2, 47, 2, 58, 2, 83, 2, 99, 2, 105, 2, 104, 2, 107, 2, 109, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 104, 2, 94, 2, 91, 2, 88, 2, 0, 0, 0, 0, 0, 0, 179, 7, 190, 7, 193, 7, 195, 7, 193, 7, 191, 7, 187, 7, 188, 7, 191, 7, 194, 7, 186, 7, 176, 7, 172, 7, 169, 7, 170, 7, 168, 7, 167, 7, 163, 7, 160, 7, 157, 7, 152, 7, 149, 7, 145, 7, 143, 7, 138, 7, 135, 7, 133, 7, 129, ......
Any help would be great. Thank you

Hi,
that's simple, just launch it via launch file. The README of kinect2_bridge states: Only available if kinect2_bridge.launch is launched.

Oops, sorry missed it
$ roslaunch kinect2_bridge kinect2_bridge.launch worked for sure
Now with $ rostopic echo /kinect2/depth_lowres/points, I have something like:
..... 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 37, 31, 32, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 41, 35, 36, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 114, 109, 110, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 98, 93, 94, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 255, 253, 254, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 255, 253, 253, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 254, 254, 254, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 192, 127, 0, 0, 0, 0, 254, 254, 254 ....
They are continuous RGB pixel elements of 960 x 540 continuous flowing images right? So I have 518400 elements for each frame? But what is the coordinate relation? I mean in .launch file it says "xyzrgb" so how can I interpret the distance? I have been searching on web but could not find any illuminating source regarding this format or conversation. Thank you

You should look at PCL documentation for information. http://pointclouds.org/documentation/tutorials/
Point clouds consists of points in space, xyz are coordinates in space and rgb is the color of that point.

Thank you so much
Best wishes

Hello, I have the similar problem.  Since I don't have a kinect at moment, I want to do the tests with rosbag (view the point cloud of the rosbag). When I'm playing the rosbag and run the kinect2_viewer, it gets stuck with the statement  "" starting receiver...""
the topic list is showed as follows:
/clicked_point
/clock
/color/sd/camera_info
/color/sd/image_color_rect
/color/sd/image_depth_rect
/initialpose
/kinect2/bond
/kinect2/depth_lowres/camera_info
/kinect2/depth_lowres/image/compressed
/kinect2/ir/camera_info
/kinect2/ir/image/compressed
/kinect2/ir_rect/camera_info
/kinect2/ir_rect/image/compressed
/kinect2/rgb_lowres/camera_info
/kinect2/rgb_lowres/image/compressed
/kinect2/sd/points
/move_base_simple/goal
/rosout
/rosout_agg
/tf
/tf_static
I think it's also lack of
/kinect2/depth_lowres/points
/kinect2/depth_highres/points
right?
Thank you very much if you could give me some help!

@summer1216 /points will be available if you launch kinect2_bridge in a new terminal. Just do "$ roscore" in a terminal and "$ roslaunch kinect2_bridge kinect2_bridge.launch" in another terminal. Then in the third terminal list the rostopics and you will see .../points.

Thanks a lot, but my corporation above is already based on kinect2_bridge.lauch. The topics from that are /kinect2/bond and /kinect2/sd/points.. And kinect2_view can not work...

ok try this in a new terminal while it roslauch still says starting receiver. "rostopic echo /kinect2/depth_lowres/points"

and try this "rosrun registration_viewer viewer kinect2 sd both" in a new terminal. Keep "roscore"rosrun kinect2_bridge kinect2_bridge" in another terminals

@summer1216 is that the full list of topics? Normally there are more topics. A complete list is given here
It seems that you are using an old version of iai_kinect2. Please pull from the current master branch and also update libfreenect2 to the latest version.

Thanks a lot @ipekbocegi  and @wiedemeyer . Now it's solved. Yes, the problem is indeed because of the version. Although I used the new version of iat_kinect2, but the topic names of the rosbag I used were according to the old version.. which the new version could not recognize. I use new recorded bags and now it's fine. Thanks!


More control over the video tag

It would be great to have the ability to store the result of the video tag in some variable.
Something similar to the images:
{% image page.photo width-400 as tmp_photo %}
https://docs.wagtail.io/en/stable/topics/images.html#more-control-over-the-img-tag
Resulting in
{% video page.video as tmp_video %}
That object should have some properties like:

thumbnail
url
transcodes
tracks
content_type

That way, it could be used as
<video
    width="{{ width }}"
    height="{{ height }}"
    {% if tmp_video.thumbnail %}poster="{{ tmp_video.thumbnail.url }}"{% endif %}>
    {% for transcode in tmp_video.transcodes %}
        <source data-src={{ transcode.url }} type="{% transcode.media_format.name %}" />
    {% endfor %}
    <source data-src={{ tmp_video.url }} type="{% tmp_video.content_type %}" />
    {% trans 'Your browser does not support the video tag.' %}
    {% for track in tmp_video.tracks %}
        {{ track.track_tag }}
    {% endfor %}
</video>

There are some cases where adding some extra properties to the source tags (ex for lazy loading) makes senses. Having a variable with all the video information would be great in those cases.

The video_tag method doesn't really do that much, so I can see this might be useful but you could achieve the same with using the Video object directly. It's useful with images because the renditions are generated using the image tag and you can access the generated url, but the video tag is just for putting all the bits together conveniently, you could still access the all the same way by doing something like:
{% with temp_video = page.video %}
... example code from above ...
{% endwith %}

Correct, thats a good solution. That's why I was adding some properties/methods in https://github.com/neon-jungle/wagtailvideos/pull/62 and https://github.com/neon-jungle/wagtailvideos/pull/63


[HOLD for payment 2024-08-09] [$250] "Leave" option icon should have "exit" icon instead of "thread" icon

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.12-0
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: @puneetlath
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1721922103779509
Action Performed:

Go to workspace list
Click 3 dot menu of the workspace where user is not an admin

Expected Result:
We should be using the exit icon for the "leave" option on workspaces
Actual Result:
Now it is a leave thread icon
Workaround:
unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos

View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01df4ffa2e8434c8ce
Upwork Job ID:<PHONE_NUMBER>583393158
Last Price Increase: 2024-07-26

Issue OwnerCurrent Issue Owner: @anmurali

Proposal
Please re-state the problem that we are trying to solve in this issue.
"Leave" option icon should have "exit" icon instead of "thread" icon
What is the root cause of that problem?
We are setting the ChatBubbles icon here
https://github.com/Expensify/App/blob/b0f810df06c3df866b7fdbed4959f8f1fef803ed/src/pages/workspace/WorkspacesListPage.tsx#L173
What changes do you think we should make in order to solve the problem?

We should change it to
                    icon: Expensicons.Exit,

What alternative solutions did you explore? (Optional)

The proposal from @daledah looks good to me!
🎀 👀 🎀 C+ reviewed!

[@mollfpr] The PR that introduced the bug has been identified. Link to the PR:

https://github.com/Expensify/App/pull/35671

[@mollfpr] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:

https://github.com/Expensify/App/pull/35671/files#r1706528957

[@mollfpr] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:

We already have the checklist for UI changes and it is still good.

[@mollfpr] Determine if we should create a regression test for this bug.
[@mollfpr] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.

Go to the workspace list
Click 3 dot menu of the workspace where the user is not an admin
Verify that: The Exit icon is displayed instead of the ChatBubbles icon
👍 or 👎

@daledah - can you pls link me your Upwork profile?

@anmurali Could you give the payment summary? Thank you!

Payment summary

[ ] @daledah - offer is here https://www.upwork.com/nx/wm/offer/103712361
[ ] @mollfpr - approved $250 on New Dot

@mollfpr feel free to request via New Dot.

$250 approved for @mollfpr


Closes: #1

Updated Validations.py python script.
Fixed the behavious of validate_user function in validations.py
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

check now.

@googlebot I signed it!

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
📝 Please visit https://cla.developers.google.com/ to sign.
Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.
What to do if you already signed the CLA
Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ Googlers: Go here for more info.

@googlebot I signed it! and we'll verify it.


WIP: End2end test p4

Description
Changes proposed in this pull request:

...
...
...

Related issue(s)

@tillknuesting please update this PR with Changes proposed in this pull request:


Generator-Liferay-Theme crashes on Node 8

When I run the generator (8.0.0-beta.1) with Node 8 it crashes with the following message:
Error: Cannot find module 'internal/util/types'
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at evalmachine.<anonymous>:31:26
    at Object.<anonymous> (/usr/local/lib/node_modules/generator-liferay-theme/node_modules/vinyl-fs/node_modules/graceful-fs/fs.js:32:1)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)

Node 8 is the current LTS version. The generator is working with Node 6.
There are a few security notices as well:
WARN notice [SECURITY] mime has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=mime&version=1.3.6 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
WARN notice [SECURITY] is-my-json-valid has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=is-my-json-valid&version=2.16.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=3.6.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.

Hey @sbaechler, I've tested this both on Windows and OSX with node v8.11.4. Both seem to work just fine.
This might be related to https://github.com/gulpjs/gulp/issues/2158. Could you try looking at that thread in case it rings a bell? The first suggestion is to run npm rebuild, or reinstalling everything.

I just tried it again with the newest version and it is working fine. Closing the issue.


Fix log write failure when installing port for the first time

When installing the port for the first time, the following code is executed before buildtrees/${PORT} is created, which results in an error Failed to open '...' for writing:
https://github.com/microsoft/vcpkg/blob/f6d52648ecdd5d57a9ca592e1425950d465aeef5/toolsrc/src/vcpkg/build.cpp#L530
So add a judgment before this code: If the The folder is created first.
Related: #10089

Hi @strega-nil, I think this PR has a high priority, please review and merge this PR first.

Linux regression unrelated with this PR.

Thanks Jack :)


Model Builder instructions for pipeline

The flow for getting a training pipeline….

Customer creates MBConfig file, trains, and generates code.
Customer checks in pipeline files based on ML Ops Solution:
a.	Azure Devops .yaml training pipeline file. Sample of pipeline generated here.
b.	Azure CLI environment .yml file. CLI sample here.
c.	Azure CLI .yml job file.
d.	Azure CLI DOCKER file
Via doc instructions, customer adds AML permissions in Azure Devops. Instructions here.
Training pipeline will now trigger when customer makes changes to mbconfig file.

There are a few options for where/how ML Ops Solution is generated:

Generate in Model Builder “Next Steps” page. There would be an option similar to “Generate pipeline code” that would create the necessary files with instructions on what to do.
File available in example repo.  An example repo would have the necessary files, with the variables needed listed at the top so customers can easily substitute their values.
Azure DevOps pipeline task, similar to these

Example instructions are in the Issues repo - https://github.com/dotnet/machinelearning-modelbuilder/tree/u/beccam/training/docs/examples


DS4 Windows Bugs out

Hi, so I'm kind of new to this dualshock to pc scene and well heres what happened
I connected both my controllers for some wii emulation action anyways they worked fine, even had custom profiles and all. But then after a few times connecting and disconnecting (over a course of a few days max) they started bugging. I mean they would connect to my pc via bluetoot but ds4 wouldn't pick them up, and then when ds4 did pick them up theyd turn off.
Well  a reinstall of ds4windows, and/or removing and reconnecting the controllers fixed it but one of my controllers glitched out I think. Well what I mean by that is its just completely shut down, it wont charge it wont connect to the pc, heck it wont even turn on. If id connect it to an external charger itd flash an orange light and that's it. If ill connect it to the pc via microusb itd just say it didn't recognize the hardware.
Now idk what to do one controllers dead in a sense, I mean the light bar wont flash, it wont charge. I know the controller isn't faulty for sure. And the microusb is fine aswell.
Any help would be amazing
Thanks!

That may because your BT adapter driver not fully installed.
Windows may install the basic BT adapter driver, but they don't provide drivers for BT service.
Most of computer's BT adapter based BCM's chip, try this driver.
I had the same issue as yours and solved by install the full BCM BT driver.
x86
https://www.driverscloud.com/en/services/GetInformationDriver/58433-327989/broadcom-btw-1201940-win8-10-x86zip
x64
https://www.driverscloud.com/en/services/GetInformationDriver/58432-327979/broadcom-btw-1201940-win8-10-x64zip

@424778940z
Thankyou, I just updated my drivers, Sorry for a month old reply btw. One controller works doesn't seem to glitch anymore, THE REAL PROBLEM is how the other controller is dead, I mean it doesn't turn on. And all It does is flash orange once, when I try putting the charger in a couple times, but that's it. Do you have any insight on that? Any help would be appreciated as a controller costs 100usd in my country and I really don't want to break my bank.

orange means it's charging, if just flash once may be the battery is full or just dead.
you have to pay attention to model numbers, here they have CUH-ZCT1U and CUH-ZCT2U
the only number in model numbers means their hardware version. ONLY version 2 have USB feature
If your dead one is version 2, try to plug in to PC see is there a new device.
If no matter what you do it still not work, get a screwdriver and open it, borrow a multimeter from someone see what voltage the battery have. a valid range should between 2.7v-4.2v. other than that means it's dead.
screwdriver and multimeter can also be found at hardware store. and you don't need an expensive multimeter to figure out the battery is dead or not, just get the cheapest one, may just $10 or less.
for replacement batteries, go ebay, i'm sure lot's of them are available


Awkward arrays in Pandas

@mhedges Do you mind if I make this a work-in-progress pull request? You can work on the branch as usual, but I can watch the developments and have a forum to ask you about it (same for you to ask me, since I'm watching this thread). I found out about the branch because it was active in Travis.

Yes, of course.  I think that’s a good idea.  Please feel free to jump in to comment or provide input.  I’m learning a lot of this as I work on it, so I would appreciate any input.
From: Jim Pivarski<EMAIL_ADDRESS>Reply-To: scikit-hep/awkward-array<EMAIL_ADDRESS>Date: Wednesday, February 27, 2019 at 10:29 AM
To: scikit-hep/awkward-array<EMAIL_ADDRESS>Cc: "Hedges, Michael T"<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: [scikit-hep/awkward-array] [WIP] Awkward arrays in Pandas (#87)
@mhedgeshttps://github.com/mhedges Do you mind if I make this a work-in-progress pull request? You can work on the branch as usual, but I can watch the developments and have a forum to ask you about it (same for you to ask me, since I'm watching this thread). I found out about the branch because it was active in Travis.

You can view, comment on, or merge this pull request online at:
https://github.com/scikit-hep/awkward-array/pull/87
Commit Summary

Copying AwkwardArray class in order to wrap in pandas ExtensionArray
[WIP] Copying AwkwardArray class in order to wrap in pandas ExtensionArray
[WIP] creating init file to load accessor, ExtensionArray, and ExtensionDType
[WIP] AwkardArray base class with pandas ExtensionArray (NDArrayOperatorsMixin in util.py)
Adding try-except for AwkwardArrayWithContent to use awkward.pandas.AwkwardArray if pandas is installed

File Changes

M awkward/array/base.pyhttps://github.com/scikit-hep/awkward-array/pull/87/files#diff-0 (5)
A awkward/pandas/init.pyhttps://github.com/scikit-hep/awkward-array/pull/87/files#diff-1 (11)
A awkward/pandas/accessor.pyhttps://github.com/scikit-hep/awkward-array/pull/87/files#diff-2 (29)
A awkward/pandas/base.pyhttps://github.com/scikit-hep/awkward-array/pull/87/files#diff-3 (457)
A awkward/pandas/util.pyhttps://github.com/scikit-hep/awkward-array/pull/87/files#diff-4 (131)

Patch Links:

https://github.com/scikit-hep/awkward-array/pull/87.patch
https://github.com/scikit-hep/awkward-array/pull/87.diff

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/scikit-hep/awkward-array/pull/87, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ARzrgH_8FjeOIo3NvhtvA6QAKIX6hB9Xks5vRrJggaJpZM4bU7x-.

Numpy's dtype=object must be avoided. It's the reason root_numpy is 40x slower than uproot (when jagged arrays are involved). If Pandas doesn't actually instantiate any dtype=object arrays but just uses the dtype to say what kinds of objects are generated on demand when you pick one row from the table, that's okay. In fact, that's why JaggedArray's dtype property is dtype(object).
Here's a way you can check: you can turn off allow_tonumpy (https://github.com/scikit-hep/awkward-array/blob/master/awkward/array/base.py#L45) to prevent the JaggedArray class instance from ever becoming a Numpy class instance. If trying to put it in Pandas raises this exception, then you'll know that Pandas is attempting to eagerly convert it into Numpy as soon as you put it in the DataFrame (bad), rather than lazily later (good). Note that letting the DataFrame print itself out would trigger whatever lazy row evaluation it night have, so this test must be performed on a DataFrame that is created and not printed.

It would be ideal if a user who never loads awkward's Pandas extension never loads Pandas, even if they have it in their system. You're deep in experimental coding, so now isn't the time to do it, but hopefully we can put the import pandas in an unimported submodule or inside methods so that the second or two of extra startup time (and the memory use) is entirely opt-in.
I saw that you have a commit saying that you moved it to awkward.util to avoid duplication, but it would be better to duplicate this bit of code than to put resource impositions on users.
But again, that's like the last problem to solve.

It's great to see work happening here! I'm rather confused by a lot of the intermediate code you've included for tests, but I understand that it's a work in progress.
It might be easier for you down the road if you git merge master earlier, rather than later. Some of the normal maintenance is touching the files you're working on, which can cause a merge headache if you let it go too far.

Thank you for the feedback!
Yeah, this is very much a work in progress and has quite a ways to go. I should note that I did not make a commit for every change I made since the vast majority of my attempts did not work.  As such, many changes were not covered by the commit messages.
I completely agree that numpy's dtype=object must be avoided.  However, I'm now actually not convinced that pandas is automatically converting to dtype=object like I originally thought.  It may only be doing that when printing the DataFrame.  I'm not sure if this is the proper metric, but when using the example awkward array in the top README file, %timeit in ipython shows the following:
%timeit df = pd.DataFrame({'array':array})
198  µs ± 2.84 µs per loop
Compare to a numpy array:
%timeit np_df = pd.DataFrame({'x':np.arange(0,10)})
224 µs ± 9.55 µs per loop
Again, I'm not sure that is the proper metric, but I'll investigate further (thanks for the tip about allow_tonumpy)!
To your last point, I think that makes sense.  I did not think of the fact that forcing the use of pandas objects could cause a performance hit.  I'll take a look after other functionality is in place , like indexing e.g. df['array'][:, :2], which is something I'm not sure how to approach yet (it currently requires the "extra" step of  df['array'].content[:, :2]).

(sorry, I thought this posted a while ago!)
Thank you for the feedback!
Yeah, this is very much a work in progress and has quite a ways to go. I should note that I did not make a commit for every change I made since the vast majority of my attempts did not work.  As such, many changes were not covered by the commit messages.
I completely agree that numpy's dtype=object must be avoided.  However, I'm now actually not convinced that pandas is automatically converting to dtype=object like I originally thought.  It may only be doing that when printing the DataFrame.  I'm not sure if this is the proper metric, but when using the example awkward array in the top README file, %timeit in ipython shows the following:
%timeit df = pd.DataFrame({'array':array})
198  µs ± 2.84 µs per loop
Compare to a numpy array:
%timeit np_df = pd.DataFrame({'x':np.arange(0,10)})
224 µs ± 9.55 µs per loop
Again, I'm not sure that is the proper metric, but I'll investigate further (thanks for the tip about allow_tonumpy)!
To your last point, I think that makes sense.  I did not think of the fact that forcing the use of pandas objects could cause a performance hit.  I'll take a look after other functionality is in place , like indexing e.g. df['array'][:, :2], which is something I'm not sure how to approach yet (it currently requires the "extra" step of  df['array'].content[:, :2]).

@mhedges, @oschulz and I were recommended to look at this PR today by Jim.  I see your last commit on this issue is April 16th.  Just checking -- are you still working on this?

Unless I am mistaken, this is now ready for testing.  I believe you will need to checkout the awk-pandas branch and run python setup.py install.
By all means, please test the interface to pandas as much as possible and let me know about any/all limitations or errors.
Note that this PR is not complete yet, as it is still lacking a way to avoid forcing all awkward-arrays to use the ExtensionArray as the base class (hence the WIP tag).

The unit tests are failing because import awkward is somewhere importing pandas and it's not installed on all systems. Somewhere, there's an import pandas that needs to be hidden within something that's not executed at startup.
We don't even want it to
try:
    import pandas
except ImportError:
    pandas = None

because that would load all of Pandas when the user might not even use it (although the user has it on their system). So this error is good: it's exposing as a test failure what would otherwise be a subtle performance bug.
One option is to hide the import pandas inside each function that uses it. The thing I like better (used by awkward-numba) is to do module-level import pandas in a module that's not automatically imported. The way your Pandas interoperability layer is working, the user has to explicitly do something to register awkward types with Pandas (just as import awkward.numba is required to register awkward types in Numba)—that thing can be a non-automatic import. Like, import awkward.pandas can be the magic command to register all those functions, and that's where the module-level import pandas can happen.

Correct, this branch currently requires awkward.util.NDArrayOperatorsMixin to take a pandas ExtensionArray as its argument, which is definitely not the long term solution.  It's an ugly hack to make it work for now until I figure out a smart way to implement what you described.
I have a couple ideas on how to do this inspired by your tip of looking at awkward-numba (thanks!).

Okay; we'll have to address that before this PR can be merged. That means that @oschulz and @wisecg can only get this new functionality by checking out the repo branch—a new version of awkward with the feature won't be available until after we've resolved this Pandas dependency.

That means that @oschulz and @wisecg can only get this new functionality by checking out the repo branch

That's fine, we can live on the (bleeding) edge for a while. More precisely, @wisecg will have to. :-) We're building a dual Julia/Python software stack, and he's handing the Python side of things (on Julia, I can stuff any vectorish-thing into a DataFrame or TypedTable).

(Yay, Julia! I actually like Julia better than Python, but Python has more market share by far. Does the Cxx.jl interface work with ROOT yet?)

Does the Cxx.jl interface work with ROOT yet?

I had that working on Julia v0.6, but on v1.x Cxx.jl was unavailable until recently. Haven't managed to find time to update ROOT.jl yet - but I definitely plan to.
Also, I recently released UpROOT.jl, wrapping scikit-hep/uproot (thanks so much for uproot, Jim!). It doesn't wrap all of uproot's functionality yet, but it's usable. With some extra Julia code, I can even decode our data files that use custom streamers.

I figured out a smarter way to implement this functionality.  Unless I am mistaken, this should be ready to merge into a release of awkward-array.
Here is a working example:
import awkward
import awkward.pandas
array = awkward.pandas.fromiter([[1.1, 2.2, None, 3.3, None],
                          [4.4, [5.5]],
                          [{"x": 6, "y": {"z": 7}}, None, {"x": 8, "y": {"z": 9}}]
                         ])
df = pd.DataFrame({'arr':array})

Printing this shows the following output:
print(df['arr'])
Out:
0                          [1.1, 2.2, None, 3.3, None]
1                                         [4.4, [5.5]]
2    [{'x': 6, 'y': {'z': 7}}, None, {'x': 8, 'y': ...
Name: arr, dtype: awkward

This works even for allow_tonumpy = False.

Actually, it would be trivial to override allow_numpy to be False for the JaggedArrayPandas class that I added here.  @jpivarski would that be worth doing?

@jpivarski in the interest of testing this new functionality more quickly, is it possible to merge michael's changes before embarking on awkward.pandas (much as I love the name) ?

@wisecg You'll find, from my example, that if you do any math on a column of the DataFrame in this state, it will lose its awkwardness. Maybe we should fix that, merge the pull request, and then start another to make awkward-pandas a PyPI package?

Thank you for the checks!  I didn't have a good intuition for how to test this, so I'm glad you found something so quickly.

So the awkward arrays need to be from awkward.pandas.* for this new way to work?

I think so.  In truth, the only requirements are that an awkward array needs to be an instance of pandas.api.extensions.ExtensionArray, and its dtype field needs to be derived from pandas.api.extensions.ExtensionDtype.  Using the awkward.pandas.* method seemed to me like a clean way to implement this.  However, I'm open to a different implementation than what I've pushed so far.

I like this approach, but it needs to be followed-through.
...
[J]ust following through with making awkward.pandas look like awkward.numba would do it.

Got it.  I'll get started.

I think this argues for awkward.pandas to be a separate package, for more complete parallelism with awkward.numba and awkward.cpp. Would you be up for that?

I was beginning to think that myself.  I'm fine with awkward.pandas being a separate package, if that is preferred.
I like @jpivarski 's suggestion to fix the math operations and merge before making a separate package.  I should be able to get this done fairly soon.

I had to think about it overnight, but now I think the implementation can be very simple. Making it parallel to awkward-numba and awkward-cpp would be a mistake because you probably want to use one or the other of those in Pandas.
All you really need are three things: (1) Pandas's Extension class to be among the supertypes of the awkward-array at all levels, (2) overridden dtypes to all say AwkwardType (let's change that to AwkwardSeries to avoid confusion with another two classes with similar names), and (3) properties like self.JaggedArray and self.Table that produce objects with the same features. This can all be done with a single mix-in and a function to apply that mix-in to existing objects. It would be a function that turns awkward-arrays into awkward-series, and then you're free to use them in Pandas.
Will you be at Fermilab today? We can probably do this together in about an hour.

Yes, I'm at Fermilab in Wilson Hall today.  I am in the 9th floor crossover area, or I can come to wherever you are.

I'd like to solve a problem I'm in the middle of right now, but other than that, I can have a free slot any time today before I have to leave at 4pm. Is there a time that would be best for you—plan on about an hour—perhaps at 11am?

11 is fine with me

Actually, could we make that 1pm? (Sorry.)

No problem!  1 pm is good with me

@jpivarski I just saw that union.py had its Pandas mixin labeled as TableSeries instead of UnionSeries, so I went ahead and changed that and pushed it into master.

@wisecg @oschulz
Our first pull request implementing awkward arrays into pandas is done!
Instead of having the pandas interface be separate, each awkward array class has a built-in way to to be put into a pandas.DataFrame by use of the array.pandas method.  Here is a working example:
import awkward
import pandas as pd
array = awkward.fromiter([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
df = pd.DataFrame({'array': array.pandas})   # Note the required use of the "pandas" method

print(df['array'])
0    [1.1 2.2 3.3]
1               []
2        [4.4 5.5]
Name: array, dtype: awkward

print(df['array'] + 100)
0    [101.1 102.2 103.3]
1                     []
2          [104.4 105.5]
Name: array, dtype: awkward

This is in the master branch now and will be included in an upcoming release.  There are a few more things that I need to add, but it should be ready for use and testing.  Please let me know how it works and if you find anything!

Thanks for fixing the UnionSeries name.
The feature is included in 0.11.0rc2 and beyond. You can get them with pip by explicitly asking for
pip install awkward>=0.11.0rc2

because you don't get pre-releases unless you ask for them. I hope to turn this (and uproot-methods and uproot) into regular, non-pre-releases before a presentation on Tuesday.  :)  (CMS ML workshop, "new features in uproot")

Our first pull request implementing awkward arrays into pandas is done!

@wisecg , go try. :-)

I can see that the code has been refactored and that the code snippet in https://github.com/scikit-hep/awkward-array/pull/87#issuecomment-502328754 above no longer works.   But, I haven't be able to see how to use the new extension setup in the pandas python file.
Can anyone post a short example?
Thanks!

That's true; the interface above was a pre-public API, the public one was announced somewhere (I can't remember where), but here it is:
>>> import awkward
>>> import pandas as pd
>>> array = awkward.fromiter([[1.1, 2.2, 3.3], [], [4.4, 5.5]])
>>> awkward.topandas(array)
0    [1.1 2.2 3.3]
1               []
2        [4.4 5.5]
dtype: awkward

In Awkward 0.x, you have to call awkward.topandas to convert a jagged array into a Pandas Series (with dtype: awkward) that Pandas can use. Of all the Awkward types, only JaggedArray has been tested in Pandas.
If you're willing to try Awkward 1.0, all array types are implicitly registered with Pandas as valid dtypes. If you
pip install awkward1    # different installation name during the transition

you can
>>> import awkward1 as ak
>>> new_array = ak.fromawkward0(array)   # convert old-style arrays into new-style
>>> new_array
<Array [[1.1, 2.2, 3.3], [], [4.4, 5.5]] type='3 * var * float64'>
>>> pd.DataFrame({"column": new_array})
            column
0  [1.1, 2.2, 3.3]
1               []
2       [4.4, 5.5]

for any Awkward type, not just arrays of lists. Also, there's a generic flattening operation that turns nested lists into MultiIndex rows and nested records into MultiIndex columns and defers to Pandas to merge different fields (corresponding to different particles, for instance) using pandas.merge's "how" parameter.
>>> ak.pandas.df(array)
                values
entry subentry
0     0            1.1
      1            2.2
      2            3.3
2     0            4.4
      1            5.5

and
>>> array2 = ak.Array([{"x": [], "y": [3, 3, 3]},
...                    {"x": [1.1], "y": [2, 2]},
...                    {"x": [2.2, 2.2], "y": [1]},
...                    {"x": [3.3, 3.3, 3.3], "y": []}])
>>> array2
<Array [{x: [], y: [3, 3, 3, ... 3.3], y: []}] type='4 * {"x": var * float64, "y...'>

>>> ak.pandas.df(array2, how="inner")
                  x  y
entry subentry
1     0         1.1  2
2     0         2.2  1

>>> ak.pandas.df(array2, how="outer")
                  x    y
entry subentry
0     0         NaN  3.0
      1         NaN  3.0
      2         NaN  3.0
1     0         1.1  2.0
      1         NaN  2.0
2     0         2.2  1.0
      1         2.2  NaN
3     0         3.3  NaN
      1         3.3  NaN
      2         3.3  NaN

That's great.  Thanks!


question: Add multiple emails to a user transform

A clear and concise description of what you want to accomplish.

I would like to add multiple emails within a user transform block. This would be useful for giving multiple email addresses the same role. Is this possible?

I would like to add multiple emails within a user transform block. This would be useful for giving multiple email addresses the same role. Is this possible?

Yes, please read https://authp.github.io/docs/authenticate/user-transforms and experiment with regex patterns. The conditional match is based on https://authp.github.io/docs/authorize/acl-rbac#conditions
There is also a way to have multiple match statements and then say match any of them. However, I don't remember how I did it. It is probably something like this.
match email1@foo
match email2@bar
default match any

I would like to add multiple emails within a user transform block. This would be useful for giving multiple email addresses the same role. Is this possible?

Yes, please read https://authp.github.io/docs/authenticate/user-transforms and experiment with regex patterns. The conditional match is based on https://authp.github.io/docs/authorize/acl-rbac#conditions
There is also a way to have multiple match statements and then say match any of them. However, I don't remember how I did it. It is probably something like this.
match email1@foo
match email2@bar
default match any

Thank you, I will try the regex patterns and report back. I was not aware of the "default match any" statement, can this be found in the docs?
Also if I would like to prevent issuance of a token unless they are matched by a user transform, how could I go about doing that. Kind of like a catch-all. Would you recommend the regex here as well, or might there be a better way?
                       transform user {
match realm google
                               no match email<EMAIL_ADDRESS>                                block
                        }

Thank you very much!

I formatted a space separated list of allowed emails like this, though I couldn't find documentation suggesting a line continuation syntax for Caddyfile.
match email \
<EMAIL_ADDRESS>\
<EMAIL_ADDRESS>

I formatted a space separated list of allowed emails like this, though I couldn't find documentation suggesting a line continuation syntax for Caddyfile.

@qrkourier , idk.
One thing that comes to mind is creating a function.
(block_user) {
  transform user {
    match realm {args[0]}
    match email {args[1]}
    block
  }
}

Then use it.
block_user google<EMAIL_ADDRESS>block_user google<EMAIL_ADDRESS>
Which would be result in:
  transform user {
    match realm google
    match email<EMAIL_ADDRESS>    block
  }

Hope this helps.

Also if I would like to prevent issuance of a token unless they are matched by a user transform, how could I go about doing that. Kind of like a catch-all. Would you recommend the regex here as well, or might there be a better way?

@sandstormkeshav , I would probably use no regex match email REGEX_PATTERN


Use unsafeFromString in Uri string literal macro

Small quality of life change - wart remover flags this with [wartremover:OptionPartial] Option#get is disabled - use Option#fold instead
Currently you have to silence it locally or turn the wart off. Hopefully a fairly harmless change?

Yeah, that seems fair.  I'll cherry-pick this to a production branch.

Cherry-picked to 0.22


"Operation not permitted" when trying using Realm in iOS Keyboard Extension with App Groups

I create iOS Keyboard Extension, I want share database between  iOS App and keyboard extension using App Groups. In iOS app I configure Realm like that:
let directory: NSURL = NSFileManager.defaultManager().containerURLForSecurityApplicationGroupIdentifier("group.myAppGroup")!
var config = Realm.Configuration()
config.path = directory.URLByAppendingPathComponent("db.realm").path
Realm.Configuration.defaultConfiguration = config

In iOS app everything work great, I can read and write to this file. But in Extension I make same config and when I try to do:
let realm = try! Realm()
I receive an error:
Optional("/private/var/mobile/Containers/Shared/AppGroup/D7E1F908-1918-4C76-8843-EEF9624E86D7/db.realm")
fatal error: 'try!' expression unexpectedly raised an error: Error Domain=io.realm Code=2 "Operation not permitted" UserInfo={NSFilePath=/private/var/mobile/Containers/Shared/AppGroup/D7E1F908-1918-4C76-8843-EEF9624E86D7/db.realm, NSLocalizedDescription=Operation not permitted, Error Code=2}: file /Library/Caches/com.apple.xbs/Sources/swiftlang/swiftlang-7<IP_ADDRESS>/src/swift/stdlib/public/core/ErrorType.swift, line 54
In iPhone Simulator everything working ! Only on real Device I have an error.
Goals
Use Real in iOS Keyboard Extension
Expected Results
Realm is working
Actual Results
Crash on init
Realm version: 0.98.6
Xcode version: 7.3
iOS/OSX version: 9.3
Dependency manager + version: Cocoapods 0.39

You should set RequestsOpenAccess to YES in Info.plist then it can access the files.

Could someone help me understand why this is necessary? I was already using app groups to share UserDefaults between my app and keyboard extension without needing RequestsOpenAccess on. The setting isn't as much of a problem as getting users to turn it on. The flow is annoying and the purpose is not obvious to users.

From Apple's documentation on App Extension Keys

RequestsOpenAccess
Boolean - iOS. Specifies whether a custom keyboard wants access to such resources as a shared container (shared with the containing app) and network access. Default value is NO. If you set this key’s value to YES, your keyboard gains a variety of capabilities:

Network access, supporting server-side processing of keystrokes
Shared container access with the keyboard’s containing app, supporting features such as an editing interface for the keyboard’s custom autocorrect lexicon
Access to Location Services and the Address Book database, with user approval
Via containing app, ability to participate in Game Center and In-App Purchase
Ability to to work with managed apps, if custom keyboard supports mobile device management (MDM)
This key is supported in iOS 8 and later.

Presumably this key is necessary to avoid exfiltrating data a user types in a keyboard without the user's knowledge.

It just seems odd that I can already share data with the host app through UserDefaults, but to share data in any other form requires this setting. Couldn't a malicious dev keylog through UserDefaults without this if they wanted?

If it behaves as you state, it does seem odd. You'd need to ask Apple why it's set up like it is.


casting a ctf with as

https://playground.solidjs.com/?hash=-2049818704&version=1.4.1

Yeah, TS expressions is not fully supported yet.

Fixed in 0.14.2


Add more functionality like symbols and math function

Adding math function would make it way more complicated for now. However, you can proceed with adding greek symbols like lambda, alpha, beta, gamma

@shristi99 What's the update?


No release branches any more?

Hi, I just can't find the release branch after V5.4.
have you guys just begin to use the master branch for releasing? just got confusing now.

Hi, I have always used the master branch as the release branch.


Crashes

When I join a new lobby or leave a lobby it crashes,
It might be because of the new update or it's just me

I noted this in release v1.2 no guarantees as to whether it will be fixed in the future, more likely than not it will be though.

I doubt this will ever work long term as they are aware of this exploit. Easy to patch.


Issues with Revert to Official Record Button

Issue 1:
Revert to Official Record button showing up for global-view
Steps to Recreate: (Global-view)

Navigate to the Select Configurations screen
Open any one of the selected Configurations
Review the screen to locate the 'Revert to Official Record' button
'Revert to Official Record' button should not be available in the Global-view

Issue 2:
Revert to Official Record button is showing up even when a MP is not checked-out
Steps to Recreate: (Workspace)

Navigate to the Select Configurations screen
Open (NOT Open & Checkout) any one of the selected Configurations
Review the screen to locate the 'Revert to Official Record' button
'Revert to Official Record' button should not be available when the MP is not checked out

Global View: