Feature Request: pass custom action buttons via slots

Currently you have support of custom action buttons via a property in component.
This is a nice feature and I use it a lot.
But it's pretty limited when you need to customize it more deeper.
For example, if I want to disable specific action button after the click or make it disable based on some condition.
Or show some buttons based on user permissions.
Ideal solution would be ability to pass whole action buttons block via named slot.
This way I will have ability to fully control all the buttons and implement my own custom logic in it.
I was trying to add my custom buttons block above the table, but then it looks not that perfect if global search is enabled.

Hi @nullday
Thanks for the valuable suggestion.
I've wrapped the existing custom action button group inside a new slot vbt-action-buttons.
Now you can use your own button group instead of the default one.
I've already made the changes the develop branch and you can use this feature from next release.
Example
<vue-bootstrap4-table :rows="rows" :columns="columns" :config="config">

    <template slot="vbt-action-buttons">
        <div class="btn-group float-right" role="group" aria-label="Basic example">
            <button type="button" class="btn btn-secondary" @click="test">Test</button>
            <button type="button" class="btn btn-secondary">Download</button>
            <button type="button" class="btn btn-secondary">Delete</button>
        </div>
    </template>

</vue-bootstrap4-table>

Note: use float-right class in the button group for good look.
Cheers,
Ruby.


Merge definitions file into opcodes

Stack:

#29
#28 ⬅
#27

⚠ Part of a stack created by spr. Do not merge manually using the UI - doing so may have unexpected results.

✓ Commit merged in pull request #29


[467] Change AccessRequest to only write UserPermission

Context

[https://trello.com/c/lj1OmlYF/467-part-5-stop-writing-user-permissions-to-organisationusers](This trello ticket)

Changes proposed in this pull request

Change AccessRequest to only write to UserPermission

Guidance to review
Checklist

[x] Make sure all information from the Trello card is in here
[x] Attach to Trello card
[x] Rebased master
[x] Cleaned commit history
[x] Tested by running locally

 (Part 5) Stop writing user permissions to organisation_users

 (Part 5) Stop writing user permissions to organisation_users

 (Part 5) Stop writing user permissions to organisation_users

 (Part 5) Stop writing user permissions to organisation_users


Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #29293

It's a good idea to open an issue first for discussion.

4 Tests pass
Appropriate changes to README are included in PR

Signed CLA


Files to track

Hi Darshan,
a small tip to improve your repository: not all kinds of files are suitable to be tracked by Git. For example, you may want to track the dictionaries of an OpenFOAM simulation, but you do not want to track the results of all the different simulations you performed. Tracking all files, including binary files, will spoil the usefulness of Git because it becomes harder to find the really important changes in files/settings. Moreover, repositories on Github have a limited size (about 1GB). There are several ways to keep your repository clean:

add files or folder to the .gitignore file such that Git ignores them; you can use regular expressions, e.g., *pdf to ignore all PDF files
use a run directory as described in the README file; the run directory is also ignored by Git

Note that once unwanted files are added to the repository, e. g., large binary files, it becomes really hard to delete them because Git is made to keep track of all changes.
Tasks

delete log files, files resulting from runtime compilation (e.g. .dep files), simulation results, ... from the repository
set up a run directory to perform simulations; if you change simulation settings, the workflow should be: 1) change the version-controlled version of the simulation, 2) copy the modfied simulation to run, and 3) re-run the simulation

Best, Andre

Hi Andre,
Thank you very much for the useful tip.  I have set a run directory as you described. I have also deleted the commit that had unnecessary(including binary) files.
Best Regards,
Darshan Thummar.


Compile errors on gcc 10.1

After upgrading mingw-gcc to 10.1.0 on Gentoo, 32-bit dxvk on master fails to compile:
build.log
Errors are all of the type
../src/util/com/../../d3d11/../dxgi/../dxvk/../vulkan/vulkan_loader.h:64:15: error: field ‘vkCreateInstance’ has incomplete type ‘dxvk::vk::VulkanFn<VkResult (__attribute__((stdcall)) *)(const VkInstanceCreateInfo*, const VkAllocationCallbacks*, VkInstance_T**)>’
   64 |     VULKAN_FN(vkCreateInstance);
      |               ^~~~~~~~~~~~~~~~
../src/util/com/../../d3d11/../dxgi/../dxvk/../vulkan/vulkan_loader_fn.h:22:28: note: in definition of macro ‘VULKAN_FN’
   22 |   VulkanFn<::PFN_ ## name> name = sym(#name)
      |                            ^~~~
../src/util/com/../../d3d11/../dxgi/../dxvk/../vulkan/vulkan_loader_fn.h:27:9: note: declaration of ‘class dxvk::vk::VulkanFn<VkResult (__attribute__((stdcall)) *)(const VkInstanceCreateInfo*, const VkAllocationCallbacks*, VkInstance_T**)>’
   27 |   class VulkanFn;
      |         ^~~~~~~~

so the problem seems to be that for some reason gcc doesn't select the correct template specialization for the Vulkan API function pointer types.
Reading through the gcc-10 changelog and porting notes, I don't see any reason why this code should fail to compile on gcc-10 but not on gcc-9, so maybe this is a gcc regression?
I had no problems compiling 64-bit dxvk.

Feel free to PR a fix. mingw isn't shipping gcc 10 yet so I can't really test that at the moment.

Doing a 32-bit build of dxvk-native with GCC 10.1 from Arch Linux staging repos works fine here.
(meson --cross-file /usr/share/meson/cross/x86-linux-gnu build)
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/10.1.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --with-isl --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-install-libiberty --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-werror gdc_include_dir=/usr/include/dlang/gdc
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.1.0 (GCC)```

@lacc97 actually can you test this: https://github.com/doitsujin/dxvk/tree/build-fix
Still no idea why the current code would fail, but if that branch works, I'll just pull that in.

Compilation works for me now on that branch, both 32-bit and 64-bit dxvk.
Yeah, it's an odd bug. It may something Gentoo-specific.

Thanks for confirming, merged the fix.

I think its related to winelib builds, this is what I'm seeing:
FAILED: src/vulkan/67fefa6<EMAIL_ADDRESS>wineg++ -Isrc/vulkan/67fefa6@@vkcommon@sta -Isrc/vulkan -I../dxvk-1.7/src/vulkan -I../dxvk-1.7/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -std=c++17 -DNOMINMAX --no-gnu-unique -O3 -m32 -msse -msse2 -fvisibility=hidden -fvisibility-inlines-hidden -D__WIDL_objidl_generated_name_0000000C= -fPIC -pthread -MD -MQ 'src/vulkan/67fefa6<EMAIL_ADDRESS>-MF 'src/vulkan/67fefa6<EMAIL_ADDRESS>-o 'src/vulkan/67fefa6<EMAIL_ADDRESS>-c ../dxvk-1.7/src/vulkan/vulkan_loader.cpp
../dxvk-1.7/src/vulkan/vulkan_loader.cpp:9:64: error: invalid conversion from ‘void (attribute((stdcall)) * ()(VkInstance, const char))()’ {aka ‘void (attribute((stdcall)) * ()(VkInstance_T, const char*))()’} to ‘PFN_vkGetInstanceProcAddr’ {aka ‘void (attribute((stdcall)) * (attribute((stdcall)) )(VkInstance_T, const char*))()’} [-fpermissive]
9 |   static const PFN_vkGetInstanceProcAddr GetInstanceProcAddr = native_vkGetInstanceProcAddrWINE;
|                                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|                                                                |
|                                                                void (attribute((stdcall)) * ()(VkInstance, const char))() {aka void (attribute((stdcall)) * ()(VkInstance_T, const char*))()}
winegcc: /usr/bin/x86_64-pc-linux-gnu-g++ failed
Reverting your removal allows 1.7 to be built and work with winelib, just passing -fpermissive stops a few things from working

I compiled dxvk-1.7.3 with gcc 10.2. on Debian Bullseye. I had to use following update alternatives:
sudo update-alternatives --config i686-w64-mingw32-gcc
sudo update-alternatives --config i686-w64-mingw32-g++
sudo update-alternatives --config x86_64-w64-mingw32-gcc
sudo update-alternatives --config x86_64-w64-mingw32-g++
and i built dxvk manually.
./package-release.sh master /your/target/directory --no-package works too.
Though dxvk-1.7.3 is ok.


Not balancing after restarting

Sometimes when stopping the Python program and restarting it afterwards, you get really sluggish balancing performance or it will fail right away. Something is slowing the loop down.
Reboot fixes it.
I have not yet found a way to reproduce this consistently. I'm opening this issue to log it when it occurs.

Today it occurred when starting it from Brickman first, and later starting it from VSCode, even though the program exited properly.

@gregcowell has reported this too.


re-write HTML report to use less JS

refer: #1270 which now is reported to not have fully resolved a performance issue for large reports.
personally I would like to use alpine-js to re-do the part that currently uses jQuery. yet to prioritize
possibly related to #2132 - best to be done at the same time cc @ericdriggs

will track this as part of #2132


Support for dotnet-tools.json?

Hello,
Can you add support for dotnet tools?
https://learn.microsoft.com/en-us/dotnet/core/tools/global-tools#install-a-local-tool
Thanks for the great extension! The default NuGet GUI does not work will with my Directory.Build.Props. NuGetMonitor works GREAT with it!

You are thinking of a command line tool?

@tom-englert Well the dotnet-tools.json has a list of tools with their nuget version:
{
  "version": 1,
  "isRoot": true,
  "tools": {
    "botsay": {
      "version": "1.0.0",
      "commands": [
        "botsay"
      ]
    },
    "dotnetsay": {
      "version": "2.1.3",
      "commands": [
        "dotnetsay"
      ]
    }
  }
}

Note "version": "1.0.0", is the NuGet package version.
If NuGetMonitor could check for this file, and check those versions against nuget, and read/write that file, that would be very helpful as I could just click a button and have my tools up to date.

Not sure if that's really fit into the scope of this tool, using a Visual Studio extension to update some global tools sounds strange.
What about writing a simple script (e.g. powershell) that calls dotnet tool list, and then for all tools calls dot tool update?

@tom-englert That is fair. I will say that global tools are not the same as tools in the dotnet-tools.json. Those are local tools, that are essentially just NuGet packages that happen to live in a json file instead of csproj or props file. Some tools I use are cake.tool, and minver. It is just as important to keep these build tools up to date as it is to keep nuget packages in a csproj or props file up to date.


Can ActiveRecord implement the interface ArrayableInterface, like Yii2?

What steps will reproduce the problem?
I overwrite the method toArray() in my Model, but when I use ArrayHelper::toArray() to convert the Model into an array, my overwritten toArray method is not called. I found that it's because ActiveRecord doesn't implement the interface ArrayableInterface
/**
 *
 * @property int $id
 * @property string|null $name
 * @property string|null $phone
 */
class User extends ActiveRecord
{
    public function tableName(): string
    {
        return '{{%user}}';
    }

    public function toArray(): array
    {
        $item = parent::toArray();
        $item['demo'] = 'demo';
        return $item;
    }
}

$user = $arFactory->createAR(User::class);
$user->name = 'xxx';

$data = ArrayHelper::toArray($user);

echo $data['demo'];

What is the expected result?
'demo'

What do you get instead?
null

Additional info

Q
A

Version
1.0.?

PHP version
8.1

Operating system
Windows 10

Thanks for reporting, you have time for a PR.


REPL : history of commands and up and down arrows

what ?

add support for arrows to see history and run past commands., just like unix
right now it is not implemented.

nvm, duplicate of #1183

Thanks for finding the dup @jalotra!


fix($animate): $animate.enabled(false) should disable animations on $animateCss as well

Closes #12685

lgtm


[W5.11][T09-A4] Marlene Koh

Enhancement:
List all persons automatically after an add or delete command

@marlenekoh Some comments added. Please close the PR after reading comments.


Update Format of AboutUs.md and Add roles

Codecov Report
Base: 72.15% // Head: 72.15% // No change to project coverage :thumbsup:

Coverage data is based on head (be321a5) compared to base (d02472c).
Patch has no changes to coverable lines.

Additional details and impacted files
@@            Coverage Diff            @@
##             master      #37   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Edit - Assigned To Function Broken

Editing the assigned to value does not work. Attempted to make a user list to display select.
As a byproduct, the save button does not work due to the assigned value being broken.

Related to issue #16

In the
router.get('/edit/:id'...)

Change:
Ticket.findByPk(req.params.id, {

to
   Ticket.findAll({
    where: {
      id: req.params.id
    },

and change:
const ticket = dbTicketData.get({ plain: true });
    res.render('edit-ticket', {
      ticket,
      loggedIn: true,
      user_username: req.session.username
    });

to
    .then(dbTicketData => {
      if (dbTicketData) {
        // const ticket = dbTicketData.get({ plain: true });
        const ticket = dbTicketData.map(ticket => ticket.get({ plain: true }));
        res.render('edit-ticket', {
          ticket,
          loggedIn: true,
          user_username: req.session.username
        });
      } else {
        res.status(404).end();
      }
    })

This changes from getting the page getting an array instead of an object. We can probably pull the full username list.

Adam made a function to fetch call the user list.


changed the energy detector to use a proxy for the output

this should fix #52, the energy detector receives and forwards the energy now completely passiv instead of actively draining energy from the input.
notes
I still get in lua sometimes miss matching results from EnergyDetector.getTransferRate() when using Mekanism universal cables, but this might be a Mekanism bug since it works now as expected when using other cables.

This looks good.
Thank you for your help. 👍🏻


poetry and private repositories

Poetry doesnt output extra index urls for having pip look at other repositories for packages by default with the export command. The --with-credentials flag must be set for the export command to put the extra index urls into the requirements.txt output. This broke my team's serverless deploy step since packaging couldn't be completed, because pip couldnt find the dependencies that were stored in our private pypi server.
I've currently worked around this issue by specifying our pypi source in PIP_EXTRA_INDEX_URL environment variable. I'm not sure this would work for people who need multiple extra sources though.
It seems an easy enough fix: use the --with-credentials flag when calling poetry export, and I'm willing to create a pr for this, unless there are repercussions to this change I'm not seeing.

Now that I think about it, I'm pretty sure adding that command would break back compat with the 1.0.0a5 release if that matters?

Hi @username-dot-com ! Thanks for offering to add that functionality.
We recently discussed poetry backwards compat in https://github.com/UnitedIncome/serverless-python-requirements/pull/446#issuecomment-587240575, and decided that we're oOK not supporting pre-1.0.0.
If you wanted to create a PR with some tests (CircleCI is the one to watch), we'd appreciate it!

Sounds good. I'm pretty busy this week, but I should be able to get on this by sometime next week.


Exposes target_system on connect(). Exposes MAVConnection.

See #400.
connect('tcp:...:...', target_system=0, target_component=100)

@tcr3dr Can you confirm that parameters are the system/component ID that we are connecting to, and that the plan is to use these when sending messages as in #400 rather than whatever value the user puts in:
 msg = vehicle.message_factory.mount_control_encode(
        0, 0,  # target system, target component

Is that correct? Are the 0 values above just ignored?
My feeling is that this isn't quite right.
I think if you have a function like the above which you are explicitly calling it should obey the system/target you enter. If we use the function internally in an exposed API (e.g. goto) then we would pass in our target_system value because this is the target that the vehicle represents.
Things are different for target_component because a vehicle represents a particular system, not a particular system and component. Do we really want to switch vehicle to talk to a camera mount (say)?  I suspect that we will need to use broadcast ID to talk to components and hope that the message gets forwarded appropriately. When sending messages "manually" as above, I would assume the coder would think about which component ID they want to direct it to, and we shouldn't override that (just advise use of the broadcast id)
Last of all, we also  want DroneKit to be able to change its own system_id so that it is possible for multiple dronekit systems to co-exist and communicate with each other. I am not sure if DK also needs to be able to set its own component ID, but let's pretend a use case exists ....
Hope some of that makes sense. As I said, I may have got this completely wrong.

@hamishwillee,
This PR now only allows the specification of the target_system ID. This is all that gets overwritten in MAVLink commands sent using the vehicle object.
If you deliver messages directly to the vehicle send_mavlink command, the target_system still gets overwritten. I'm unsure what to do here; I feel like we should expect the target_system to be fixed to that of the vehicle it's being called on. If we want to communicate with other target_systems, it should be done through (perhaps) another interface?
We should support the changing of our own system's system_id, but I am not sure how that logic works yet, and seems like it can be an enhancement.

@tcr3dr Isn't the target_system id the ID of the system you want to talk to? In that case, wouldn't you be better off querying that from the vehicle you've connected to rather than trying to set it in connect?. I presume this is what we do now but maybe we just always broadcast.
Once we have the value, however we get it, then yes, we should use it for our internal message sending since vehicle represents a specific vehicle.
My (possibly idiotic) take on communicating with other systems is that if you want to do this you should create a separate vehicle object and handle any inter-vehicle communication in your script. The implication of this is

Yes, the target_system_id should be used for all communication from this vehicle object (overwriting whatever might be set in send_mavlink.
IMO if we aren't allowing people to set this parameter in send_mavlink, we should probably remove it from the public API (and if we call the same API internally, we'll need both an internal and external message sending API)

Yes, changing our own system_id is an enhancement. I don't think this is complicated - and in most cases it is something you wouldn't need to do. I'll leave you to create a separate issue because I'm a little lazy!

@tcr3dr
I still think it doesn't make much sense to allow people to specify the target_system - this should be obtained from the connected vehicle.
That said, can you confirm that target_system parameter is only thing requiring docs from this?

@hamishwillee See #458. It makes sense in that we actually have a manual override in the Solo code—also this is the first step to implementing it as an automatic process.
target_system is the only thing requiring docs. I may or may not be exposing MAVConnection but it probably involves another PR and me writing up some examples about that. For now, ignore.

@tcr3dr Does this also set the target_system used when you set a Command (in a waypoint)?


Issue when changing the page size via dropdown

The issue is that the "1-10 displayed" text doesn't get reported correctly on pages 2+ after changing the page size. For example lets say we have 60 items and on the initial page load paging is set to 5 items. It will display 1-5, then click page 2 it displays 6-10. It works how we expect; However, when we use a dropdown to change the number of items to 10 per page. Page 1 displays 1-10, but page 2 displays 11-60 as the text, when it should display 11-20.
See this plunker for the issue.
After making the plunker I see that the page size jumps on page 2 and displays more items than it is suppose to. When I first discovered this I was using sever side queries, so my pages would at least retain the correct number of items.

This is a classic angular problem when binding to a select.  Here's a way of solving this problem:
http://plnkr.co/edit/zNGFCUqoA9W4VHFMYnGk?p=preview
...copy pasted from
https://docs.angularjs.org/api/ng/directive/select


[T2A3/A4][F10-A2]Jeremy Goh

Both A3 and A4 are included in this request

You should separate T2A3 and T2A4 as 2 different pull request :)


Chap 9, p.248, California housing dataset: "BadZipFile: File is not a zip file"

Hi, I'm getting an error message when I try to run this code on Chapter 9, p.248.  Any suggestions for solving this please or is the resource link broken?
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
m, n = housing.data.shape
housing_data_plus_bias = np.c_[np.ones((m, 1)), housing.data]
The final error message is
BadZipFile: File is not a zip file

Hi @dpnolan ,
That's odd, I just tried it and everything worked fine.  Perhaps it's a temporary connectivity issue or server error? Could you please retry?
Here's my output:
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sklearn.datasets import fetch_california_housing
>>> housing = fetch_california_housing()
downloading Cal. housing from http://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.tgz to /home/ageron/scikit_learn_data
>>> m,n=housing.data.shape
>>> housing_data_plus_bias = np.c_[np.ones((m,1)), housing.data]
>>> housing_data_plus_bias
array([[   1.        ,    8.3252    ,   41.        , ...,    2.55555556,
          37.88      , -122.23      ],
       [   1.        ,    8.3014    ,   21.        , ...,    2.10984183,
          37.86      , -122.22      ],
       [   1.        ,    7.2574    ,   52.        , ...,    2.80225989,
          37.85      , -122.24      ],
       ...,
       [   1.        ,    1.7       ,   17.        , ...,    2.3256351 ,
          39.43      , -121.22      ],
       [   1.        ,    1.8672    ,   18.        , ...,    2.12320917,
          39.43      , -121.32      ],
       [   1.        ,    2.3886    ,   16.        , ...,    2.61698113,
          39.37      , -121.24      ]])

Notice the URL where Scikit-Learn fetches the data. By default, this tgz file is saved to ~/scikit_learn_data:
$ ls -l ~/scikit_learn_data/
total 372
-rw-r--r-- 1 ageron ageron 366877 avril 16 22:37 cal_housing_py3.pkz
drwxr-xr-x 2 ageron ageron   4096 avril  8 14:21 mldata

Perhaps this file exists on your machine, but it is corrupted? If so, try deleting it, then try again.

Hi Aurélien, I have this working on the Python 2 version when I run that. There seem to be some problems with my Anaconda setup, so I will re-install. I should perhaps not start off with chapter 9.  Thanks.  It's a great book.  Peter

@dpnolan I'm really glad you enjoy the book, thanks for your feedback and good luck with your installation.


Ranking genes between specific clusters in clonotype network

Dear ICBI lab,
I have some questions regarding the Scirpy package, I would really appreciate your help.
For my analyses I first merged TCR and transcriptomics data from two different samples (organ 1 and organ 2), subsequently I then merged these two files.
1. a) If using the Scirpy clonotype_network tool the 'sequence' is set to 'nt', are the clusters in the clonotype network (where each node represents a cell) based on identical nucleotide sequences, or is it based on similarity, meaning that one cluster could consists of cells with slightly different nucleotide sequences?
As I understand, each node represents a cell, and edges connect cells belonging to the same clonotype. The function makes visualization of the clonotype-network possible, analogous to the construction of a neighborhood graph from transcriptomics data with the Scanpy package, so that based on the above, it computes a neighborhood graph of CDR3 nucleotide sequences with "scirpy.pp.tcr_neighbors())". However answer to my question I couldn't find, I was hoping you can help me out?
b) Do the lines that connect the cells within a cluster have any meaning?

c) In the plot above, is it correct that the closer the different clusters are together, the more similar their nucleotide sequences are? Meaning that the sequences of the clonotypes consisting of only 2 cells in this case are most different from the clonotype clusters consisting of >5 cells (as they are further apart)?
2. a) The package allows for specifying what organs the clusters consist of:

My question is: how can I select specific clusters in the clonotype network graph (I only would like to include the clonotypes in the network that have identical clonotypes between the two different samples from my data, i.e. organ 1 and organ 2), meaning that in the plot above I would like to filter for the clusters only that display both blue and orange nodes.
b) How can I assign numbers to my clusters based on identical nucleotide sequences shared between two samples?
c) How can I add a legend to my plot, and how can I change the name 'batch' into 'sample'?
3. Using Scirpy, how can one best specify differentially expressed genes (based on the transcriptomics data) between the different clusters based on shared nucleotide sequences between blood and fat, for example cluster 1 vs. rest of the clusters, and cluster 1 and 2 vs. cluster 3 and 4? I have tried implementing Scanpy's tool to rank genes using WIlcoxon, but unfortunately I can't make It work.
sc.tl.rank_genes_groups(adata, 'clonotype', groups=['1','2'], reference=['3','4'], method='wilcoxon')
sc.pl.rank_genes_groups(adata, groups=['1', '2'], n_genes=20)
-The key of the observations grouping to be considered would be:
"clonotype clusters based on share identical nucleotide sequences between organ 1 and organ 2".
-Subset of groups to which comparison wouldl be restricted: "clonotype 1 and clonotype 2"
-Comparison: Compare with respect to a specific group
-Group identifier with respect to which compare: "clonotype 3 and clonotype 4"
“The number of genes that appear in the returned tables”: 100
“Method”: Wilcoxon-Rank-Sum
Thanks in advance,
Josine

Hi Josine,

If using the Scirpy clonotype_network tool the 'sequence' is set to 'nt', are the clusters in the clonotype network (where each node represents a cell) based on identical nucleotide sequences

If you use the default options (i.e. don't specify the metric parameter) it will be based on sequence identity.

b) Do the lines that connect the cells within a cluster have any meaning?

an edge means "these two cells have the same clonotype" (when using identity metric), or "these two cells are in the same clonotype cluster" (when using a similarity metric). When using an identity metric, clusters are always fully connected. This is not necessarily the case when using a similarity metric.

In the plot above, is it correct that the closer the different clusters are together, the more similar their nucleotide sequences are? Meaning that the sequences of the clonotypes consisting of only 2 cells in this case are most different from the clonotype clusters consisting of >5 cells (as they are further apart)?

No, the distance has no meaning. This is just an artifact from the layouting algorithm.
When using a similarity metric, edges are colored by the distance, but the point can be made that it's hard to see (one could experiment with the edges_cmap parameter).

how can I select specific clusters in the clonotype network graph

You can just subset the AnnData object, e.g.
adata_subset = adata[adata.obs["clonotype"].isin(["23", "42"]), :]
ir.tl.clonotype_network(adata_subset)
ir.pl.clonotype_network(adata_subset)

How can I assign numbers to my clusters based on identical nucleotide sequences shared between two samples?

You just need to find out which clonotypes have more than two samples. There are many ways to do this, here's an (untested) approach using pandas. You can then subset adata as described above.
samples_per_clonotype = adata.obs.groupby("clonotype").apply(lambda group_df: len(set(group_df["batch"])))
clonotypes_with_two_or_more_samples = samples_per_clonotype[samples_per_clonotype > 2].index

How can I add a legend to my plot, and how can I change the name 'batch' into 'sample'?

Set legend_loc="right margin". The easiest way to change the title is to rename the column. Alternatively, you can retrieve the matplotlib.Axes object and call set_title. This is also the way to go if you further want to customize the plot.

Using Scirpy, how can one best specify differentially expressed genes (based on the transcriptomics data) between the different clusters based on shared nucleotide sequences between blood and fat, for example cluster 1 vs. rest of the clusters, and cluster 1 and 2 vs. cluster 3 and 4? I have tried implementing Scanpy's tool to rank genes using WIlcoxon, but unfortunately I can't make It work.

This is exactely how it should work. There's also an example in the tutorial. Do you get any error message?
Cheers,
Gregor

Dear Gregory,
Thanks so much for your reply. All is clear to me now, except from the last part. I did see the (last) example in the tutorialhttps://icbi-lab.github.io/scirpy/tutorials/tutorial_3k_tcr.html, which is what I'd need (partly), however:

When I plot my clonotype network it doesn't show the number of the clonotype, so how can I determine which ones I have to select, what name (in tutorial it is "163_TCR" and "277_TCR")?
How can I plot differentially expressed genes between the two batches (samples) within one clonotype? I was able to plot which clonotypes look like they consist of both samples, see second graph where orange is organ 1, and blue is organ 2 (again, I don't know what number they have though).

Thanks 🙏 🙏
Josine
[cid:b9aad2ae-35bc-46a3-80c6-c11be8a940c9]
[cid:23ec743b-1e97-43e6-a7d8-75d18e9af1ad]
Josephina Jansen, MD
Postodoctoral Associate
Yale School of Medicine | Department of Neurology
Combined Utrecht University PhD Program | Clinical and Experimental Neuroscience
<EMAIL_ADDRESS>Phone:<PHONE_NUMBER> |<PHONE_NUMBER>

From: Gregor Sturm<EMAIL_ADDRESS>Sent: Thursday, October 15, 2020 5:31 AM
To: icbi-lab/scirpy<EMAIL_ADDRESS>Cc: Jansen, Josephina<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [icbi-lab/scirpy] Ranking genes between specific clusters in clonotype network (#204)
Hi Josine,
If using the Scirpy clonotype_network tool the 'sequence' is set to 'nt', are the clusters in the clonotype network (where each node represents a cell) based on identical nucleotide sequences
If you use the default options (i.e. don't specify the metric parameter) it will be based on sequence identity.
b) Do the lines that connect the cells within a cluster have any meaning?
an edge means "these two cells have the same clonotype" (when using identity metric), or "these two cells are in the same clonotype cluster" (when using a similarity metric). When using an identity metric, clusters are always fully connected. This is not necessarily the case when using a similarity metric.
In the plot above, is it correct that the closer the different clusters are together, the more similar their nucleotide sequences are? Meaning that the sequences of the clonotypes consisting of only 2 cells in this case are most different from the clonotype clusters consisting of >5 cells (as they are further apart)?
No, the distance has no meaning. This is just an artifact from the layouting algorithm.
When using a similarity metric, edges are colored by the distance, but the point can be made that it's hard to see (one could experiment with the edges_cmap parameterhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Ficbi-lab.github.io%2Fscirpy%2Fgenerated%2Fscirpy.pl.clonotype_network.html%23scirpy.pl.clonotype_network&data=02|01|josephina.jansen%40yale.edu|29850f5cb43d4344859c08d870ed2abb|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637383511218813579&sdata=AvqLZI1jW0CCSQWv71u72yhUx39RAvo8Zrb%2BXe2T8Ow%3D&reserved=0).
how can I select specific clusters in the clonotype network graph
You can just subset the AnnData object, e.g.
adata_subset = adata[adata.obs["clonotype"].isin(["23", "42"]), :]
ir.tl.clonotype_network(adata_subset)
ir.pl.clonotype_network(adata_subset)
How can I assign numbers to my clusters based on identical nucleotide sequences shared between two samples?
You just need to find out which clonotypes have more than two samples. There are many ways to do this, here's an (untested) approach using pandas. You can then subset adata as described above.
samples_per_clonotype = adata.obs.groupby("clonotype").apply(lambda group_df: len(set(group_df["batch"])))
clonotypes_with_two_or_more_samples = samples_per_clonotype[samples_per_clonotype > 2].index
How can I add a legend to my plot, and how can I change the name 'batch' into 'sample'?
Set legend_loc="right margin". The easiest way to change the title is to rename the column. Alternatively, you can retrieve the matplotlib.Axes objecthttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmatplotlib.org%2F3.1.1%2Fapi%2Faxes_api.html&data=02|01|josephina.jansen%40yale.edu|29850f5cb43d4344859c08d870ed2abb|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637383511218823568&sdata=ZQhAxjBlEjDqts%2F54BTEmNS22d%2Bnxp%2FaJKKdyjkYh1U%3D&reserved=0 and call set_title. This is also the way to go if you further want to customize the plot.

Using Scirpy, how can one best specify differentially expressed genes (based on the transcriptomics data) between the different clusters based on shared nucleotide sequences between blood and fat, for example cluster 1 vs. rest of the clusters, and cluster 1 and 2 vs. cluster 3 and 4? I have tried implementing Scanpy's tool to rank genes using WIlcoxon, but unfortunately I can't make It work.

This is exactely how it should work. There's also an example in the tutorialhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Ficbi-lab.github.io%2Fscirpy%2Ftutorials%2Ftutorial_3k_tcr.html%23Marker-genes-in-top-clonotypes&data=02|01|josephina.jansen%40yale.edu|29850f5cb43d4344859c08d870ed2abb|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637383511218823568&sdata=hLGoxDC9fc%2FJPZn2RjJLmMdL1fu3Y%2Bci965atHKeqh0%3D&reserved=0. Do you get any error message?
Cheers,
Gregor
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ficbi-lab%2Fscirpy%2Fissues%2F204%23issuecomment-709038904&data=02|01|josephina.jansen%40yale.edu|29850f5cb43d4344859c08d870ed2abb|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637383511218833564&sdata=8QwH0MUX9ebHwXHedN5tC0jfoI35ITGeghBimAOHaQs%3D&reserved=0, or unsubscribehttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQ355Z4PQQSSIZSJIFRY3F3SK26Q5ANCNFSM4SRNZV6A&data=02|01|josephina.jansen%40yale.edu|29850f5cb43d4344859c08d870ed2abb|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637383511218833564&sdata=sZ9mtUyLMVJIQynhptvrfRFBA%2BP8zSe24Q8IIFZBT5w%3D&reserved=0.

Hi Josine,
glad I could help :)

When I plot my clonotype network it doesn't show the number of the clonotype, so how can I determine which ones I have to select, what name (in tutorial it is "163_TCR" and "277_TCR")

You'll have to plot the clonotype network colored by clonotype and with legend_loc="on data".
ir.pl.clonotype_network(adata, color="clonotype", legend_loc="on data")

Actually, it would be nice to show overlay the clonotype number also if colored by other categories, I'll consider implementing that.

How can I plot differentially expressed genes between the two batches (samples) within one clonotype?

Once you found out the clonotype number, you can again subset adata and perform differential expression using scanpy:
adata_subset = adata[adata.obs["clonotype"].isin(["23", "42"]), :]

Let me know if there are further issues!

Best,
Gregor
sc.tl.rank_genes_groups(adata_subset, 'batch', method="wilcoxon")

Dear Gregor,
I can overlay the clonotype number, but when I try to plot it the second and third line here give an error:

Dear Gregor,
The scripts seem to throw some errors, could you perhaps point me into the right direction once more?
I can now overlay the clonotype number, but when I try to plot a selection of clonotypes, there is an error:

adata_subset_tumor = adata_tumor[adata_tumor.obs["clonotype"].isin(["25"]), :]
ir.tl.clonotype_network("adata_subset_tumor")
ir.pl.clonotype_network("adata_subset_tumor")

Error: 'str' object has no attribute 'uns'
The following code gives another error:

sc.tl.rank_genes_groups(adata_subset_tumor, 'batch', method='wilcoxon')
sc.pl.rank_genes_groups(adata_subset_tumor, n_genes=25, sharey=False)

Error: 'division by zero'.
Then if I want to rank DE genes between certain clonotypes, I was thinking to make two subsets, but here I also get an error:

_final_data_subset = final_data[final_data.obs["clonotype"].isin(["25"]), :]
final_data_subset2 = final_data[final_data.obs["clonotype"].isin(["110", "112"]), :]

sc.tl.rank_genes_groups(final_data_subset, 'final_data_subset2', method='wilcoxon')

sc.pl.rank_genes_groups(final_data_subset, n_genes=25, sharey=False)_

Error: 'final_data_subset2'
How to solve these three errors?
Appreciatively,
Josine

Dear Gregor,
Ranking the DE genes between specific clonotypes now works perfectly, as does ranking DE genes within one clonotype that is shared between two organs, so first of all: thank you!!!
My other scripts seem to throw some errors, could you perhaps point me into the right direction once more?
When ranking DE genes between different subsets of clonotypes, I create multiple subsets, as suggested by your previous comment, for example:

final_data_subset = final_data[final_data.obs["clonotype"].isin(["100"]), :]
final_data_subset2 = final_data[final_data.obs["clonotype"].isin(["908", "9775", "2488", "501"]), :]

Unfortunately the following lines then throw an error:

sc.tl.rank_genes_groups(final_data, "clonotype", groups=["final_data_subset"], reference="final_data_subset2", method="wilcoxon"
)
sc.pl.rank_genes_groups_violin(final_data, groups="final_data_subset", n_genes=15)

ValueError: reference = final_data_subset2 needs to be one of groupby = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', etc.]
I subsequently tried the following, but there was another error:

sc.tl.rank_genes_groups(final_data, "clonotype", groups=[“100”], reference= (“908", "9775", "2488", "501"), method="wilcoxon")
sc.pl.rank_genes_groups_violin(final_data, groups=“100”, n_genes=15)

**ValueError: reference = ('848', '996') needs to be one of groupby = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13’,] etc.
Not sure what I am doing wrong, how can I best solve these errors?**
Appreciatively,
Josine

Hi Josine,
the first one doesn't work, because you are defining two variables (final_data_subset, final_data_subset2) which are essentially AnnData objects on their own. They are not stored in final_data. Therefore, rank_gene_groups(final_data, ...) can't know about them.
I initially thought your second approach should work, but apparently, scanpy only accepts a single group as reference, not a list.
To make this work, I would create an additional column in adata.obs containing your annotation. One way to do that is using pandas indexes (see below). But you could also use a loop or list-comprehension to build the column.
# initialize the whole column with 'other'
adata.obs["my_groups"] = "other"
# set the corresponding rows in the "my_groups" column to 'group1'.
# .loc works like this: .loc[ ROW_LABELS, COL_LABELS ]
adata.obs.loc[adata.obs["clonotype"].isin(["908", "9775", "2488", "501"]), "my_groups"] = "group1"
adata.obs.loc[adata.obs["clonotype"].isin(["100"]), "my_groups"] = "group2"

Then you can do
sc.tl.rank_genes_groups(adata, "my_groups", groups="group1", reference="group2", method='wilcoxon')

Best,
Gregor
P.S.:  you can format code in github using backticks:
```python
[your code here]
```

Topic: creating a subset from a subset
Works now! I was able to manually filter for the clonotypes of interest (ones that only occur in organ 1 & organ 2 = subset1), and plot this subset, and then rank genes between different clonotypes, groups of clonotypes.
I did this based on the following figures:

Also DE genes between two batches within a clonotype worked, thanks!
Then, I wanted to make two other subsets to only check for DE genes between the organ 1 cells that are part of subset 1 (which should create subset 2), and the organ 1 cells that belong to the NON-clonotypes/or singletons (constructing subset 3).
I think subset 2 would need to be:
[adata_subset2 = adata_subset1[adata_subset1.obs["batch"].isin(["blood"]), :]

### But how to fix a code for subset 3?
I tried the following 2 methods (none of which works):

Trying to select that same way as I did for subset 1 based on the following plots (you may see this is impossible, also when changing palette and/or panel_size):

[ir.tl.clonotype_network(adata, sequence='nt', **min_size=1**)
ir.pl.clonotype_network(adata, color="batch", panel_size=(10, 10), legend_loc="none") ]

[# Then overlay with number so to create subset 3.
ir.tl.clonotype_network(final_data, sequence='nt', min_size=1)
ir.pl.clonotype_network(final_data, title='Clonal Expansion incl. singletons',  color="clonotype", panel_size=(200, 200), legend_loc="on data") ]

And zoomed in:

Making a subset that only includes singletons - again, manually based on plotting the clonotype_network, after which I would, just like for subset 2, select for "batch"].isin(["blood"]), :]  (this would construct subset 3). But, apparently, when computing the clonotype_network, a max_size is not an option, and throws an error:

[ir.tl.clonotype_network(final_data, sequence='nt', max_size=2)]

The above methods obviously don't work. Would you have any suggestions to code for subset 3 (is that possible at all)? Then I can compare subset 2 and 3, based on your earlier advice.
So just to be clear: the only thing that is not working is my subset 3 which should be:  all non-clonotypes in organ 1
Many, many thanks (again) 🙈😀😊

Topic: DE genes between organ 1 cells belonging to the clonotypes that have identical CDR3 sequences in both organ 1 and 2 & NON-clonotypes in organ 1
So subset 1 now only consists of clonotypes that have a shared cdr3_nt sequence in organ 1 as well as in organ 2.
I would like to make two other subsets to only check for DE genes between the organ 1 cells that are part of subset 1 (which should create subset 2), and the organ 1 cells that belong to the NON-clonotypes/or singletons (constructing subset 3).
I think subset 2 would need to be:
Python[[adata_subset2 = adata_subset1[adata_subset1.obs["batch"].isin(["blood"]), :]]
But how to fix a code for subset 3?
I tried the following 4 methods (none of which works):

Trying the same thing as I did for subset 1, but manually selecting the appropriate clonotypes is impossible, as when I display the clonotypes min_size = 1 (because those are the ones I want for subset 3 displays an unreadable graph (also when changing palette and/or panel_size)

Overlaying with number so to select the ones for subset 3:

Making a subset that only includes singletons - again, manually based on plotting the clonotype_network, after which I would, just like for subset 2, select for "batch"].isin(["blood"]), :] (this would construct subset 3). But, apparently, when computing the clonotype_network, a max_size is not an option, and throws an error:

python[ir.tl.clonotype_network(final_data, sequence='nt', max_size=2)]

Then I tried unfortunately threw the following error: KeyError: 'TRB_1_cdr3_nt' , even though my adata definitely has the obs. TRB_1_cdr3_nt.

python[adata_NonClonotypes= adata[adata['TRB_1_cdr3_nt'].isin(adata['TRB_1_cdr3_nt'].value_counts()[adata['TRB_1_cdr3_nt'].value_counts()==1].index)] ]

The last thing I tried was to add this as an obs to my object, so I could make use of the  adata.obs["my_groups"] = "other", and setting group 1 and 2 etc.

python[adata.obs['adata_NonClonotypes'] = adata[adata.obs["TRB_1_cdr3_nt"].isin(adata.obs['TRB_1_cdr3_nt'].value_counts()[adata.obs['TRB_1_cdr3_nt'].value_counts()==1].index)]
python[adata.obs['adata_subset_blood_from_NONclonotypes'] = final_data[final_data.obs["TRB_1_cdr3_nt"].isin(final_data.obs['TRB_1_cdr3_nt'].value_counts()[final_data.obs['TRB_1_cdr3_nt'].value_counts()==1].index)]]
ValueError: 'Length of values does not match length of index'
The above methods obviously don't work. Would you have any suggestions to code for subset 3 (is that possible at all)? Then I can compare subset 2 and 3, based on your earlier advice.
So just to be clear: the only thing that is not working is my subset 3 which should be: all non-clonotypes in organ 1
I assume there is no function: "scirpy.til.define_NON-clonotypes()"?
Many, many thanks (again) 🙈😀😊

One more question:
When I run the following code, the intended graphs are displayed.
final_data_subsetX = final_data[final_data.obs["clonotype"].isin(["42"]), :]
sc.tl.rank_genes_groups(final_data_subsetX, 'batch', method="wilcoxon")
sc.pl.rank_genes_groups(final_data_subsetX, n_genes=25, sharey=False)
It works fine for some other clonotypes, too.
But when I enter (Murgphy's law) the one I need, (of which I am certain it is in adata), there is a ZeroDivisionError.
How to solve this?
Example: this one works fine:

Hi Josine,
you are on the right track with making a column (as unfortunately can't do the DE comparison between subsets with scanpy). However, there are a few (pandas) problems with your steps, I'll try to explain what's wrong:

adata.obs["subset2"] = "adata_subset_organ1_from_clonotypes"

Like that you just assign a string ("adata_subset_organ1_from_clonotypes") to all rows of adata.obs. To a column, you can either assign a constant value (like here) or an array (or list) with the same length as the adata.obs has rows.
Therefore, leaving away the quotes would lead to an error.

adata_NonClonotypes= adata[adata.obs["TRB_1_cdr3_nt"].isin(adata.obs['TRB_1_cdr3_nt'].value_counts()[adata.obs['TRB_1_cdr3_nt'].value_counts()==1].index)]

You can have that easier by using the clonotype_size column ;)

adata.obs.loc[adata.obs["subset2"].isin(["organ1"]), "my_groups"] = "group2"

Here you assign group2 to those rows, where the column subset2 == "organ1". This works in principle, but
I think you don't have the right values in the subset2 columns.
Alternatively, you can use the cell indexes from the subset you created to assign the values:
adata.obs.loc[adata_subset_organ1_from_clonotypes.obs_names, "my_groups"] = "group2"`

Maybe this tutorial could help you to familiarize yourself with the pandas indexes:
https://www.shanelynn.ie/select-pandas-dataframe-rows-and-columns-using-iloc-loc-and-ix/

I think a short version of what you're trying to achieve could look like this:
# Step 1: create "boolean masks", i.e. arrays with the same length as adata.obs
# that indicate if a certain row meets a certain condition
is_organ1 = (adata.obs["batch"] == "organ1")
is_singleton = (adata.obs["clonotype_size"] == 1)
# (or just use directly what you used to create subset1)
is_subset1 = adata.obs_names.isin(adata_subset1.obs_names)

# Step 2: create a column using the masks. & means boolean
# and, i.e. decide for each row if both conditions are true
adata.obs["new_col"] = "other"
adata.obs.loc[is_subset1 & is_organ1, "new_col"] = "subset1+organ1"
adata.obs.loc[is_organ1 & is_singleton, "new_col"] = "singleton+organ1"

Then you can perform the wilcox test as you already know.

When I run the following code, the intended graphs are displayed for some clonotypes, but not for others.
adata_subsetX = adata[adata.obs["clonotype"].isin(["Number"]), :]
sc.tl.rank_genes_groups(adata_subsetX, 'batch', method="wilcoxon")
sc.pl.rank_genes_groups(adata_subsetX, n_genes=25, sharey=False)

I suspect this happens if there are too few/no cells in a subset and no meaningful ranks can be computed for the genes. I would check the gene expression of these groups manually (e.g. using a scanpy matrixplot) to get an idea.

Thanks so much for your help/suggestions, they have been very useful and I can apply them now for my other analyses!

Question: how can I save only the annotated data from one batch (organ_1) within a clonotype (number 1116) to csv?
Clonotype1116 = adata[adata.obs["clonotype"].isin(["1116"]), :]
I think the annotated data belonging to clonotype 1116 can be saved like this:
Clonotype1116.obs.to_csv('Filename', sep=',')
But the following does not work, which I can understand (AttributeError: 'DataFrame' object has no attribute 'obs'):
File_Organ1 = Clonotype1116.obs.loc[Clonotype1116.obs["batch"].isin(["tumor"])]
FIle_Organ1.obs.to_csv('Filename', sep=',')
How can the annotated data belonging to batch = Organ 1 from clonotype 1116 be saved?

This sounds like you try to get an obs from an obs
# df is a dataframe
df = adata.obs

# error!
df2 = df.obs

# but you can save a dataframe directly
df.to_csv()

On Tue, Nov 10, 2020, 05:18 Josine<EMAIL_ADDRESS>wrote:

Question: how can I save only the annotated data from one batch (organ_1)
within a clonotype (number 1116) to csv?
Clonotype1116 = adata[adata.obs["clonotype"].isin(["1116"]), :]
I think the annotated data belonging to clonotype 1116 can be saved like
this:
Clonotype1116.obs.to_csv('Filename', sep=',')
But the following does not work, which I can understand (AttributeError:
'DataFrame' object has no attribute 'obs'):
File_Organ1 =
Clonotype1116.obs.loc[Clonotype1116.obs["batch"].isin(["tumor"])]
FIle_Organ1.obs.to_csv('Filename', sep=',')
How can the annotated data belonging to batch = Organ 1 from clonotype
1116 be saved?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/icbi-lab/scirpy/issues/204#issuecomment-724439711,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVZRVYEJQ533HPSBM3ABO3SPC5K3ANCNFSM4SRNZV6A
.

Thanks, I see. Saving the df directly works in general, but not when trying to save only the data belonging to cells in a specific batch within a clonotype. The following error then occurs: "AttributeError: 'DataFrame' object has no attribute 'write_csvs'"
Saving data belonging to for example one clonotype works:
df1 = adata[adata.obs["clonotype"].isin(["1116"]), :]
df1.write_csvs('file_name1', sep=',')
Saving annotated data from all cells from the batch 'organ1' within clonotype '1116' does not work, however that's what I'd like to end up with:
df2 = adata.obs.loc[df1.obs["batch"].isin(["blood"]), :]    # This works fine
df2.write_csvs('file_name2', sep=',') # This is the line throwing the error.

It's a detail ;)
write_csv is defined on a DataFrame, i.e. on adata.obs.
write_csvs is defined on AnnData, i.e. on adata directly.
The former will only save the obs data frame, the latter will write both obs and var data frames (and potentially others) stored in an AnnData object.

OH, 🙈THANK YOU!!!

Hi again, so I have tried many things now, but I am not proceeding with the following:
ranking genes between selection A of clonotypes --> clonotypes only occurring in the brain ("is_brainOnly") and selection B --> only batch= blood from selection B ("is_shared"). Selection B consists of clonotypes that occur both in brain AND in blood.
What am I doing wrong:
As you suggested, here is step 1 to create boolean masks
**is_brain** = (adata.obs["batch"] == "brain")
is_shared = adata[adata.obs["clonotype"].isin(["1527","1611","5502","1494","7112","296","3399","9309","8951","863","1991","3461","6214","5086","5864","3605","6286","3892","996","2488","100","6123","7317","5417","7279","2596","6527","501","1688","287","2984","3033","1116","2574","76","5250","1506","1209","1445","848","42"]) , :]
is_brainOnly = final_data.obs.loc[final_data.obs["clonotype"].isin(["10231", "10491", "9803", "10698", "10084", "9447", "9448", "10362", "10492", "9641", "11010", "10584", "11483", "10061", "10015", "10036", "10437", "10459", "9775", "9784", "9812", "9919", "9465", "9599", "9647", "9642", "9916", "9830", "9450", "10526", "10985", "10717", "11279", "9453"]), :]
Step 2: create a column using the masks and decide for each row if both conditions are true
adata.obs["new_col"] = "other"
adata.obs.loc[adata.obs["clonotype"].isin(["10231", "10491", "9803", "10698", "10084", "9447", "9448", "10362", "10492", "9641", "11010", "10584", "11483", "10061", "10015", "10036", "10437", "10459", "9775", "9784", "9812", "9919", "9465", "9599", "9647", "9642", "9916", "9830", "9450", "10526", "10985", "10717", "11279", "9453"]), "new_col"] = **"group1"**
adata.obs.loc[is_shared & **is_brain**), "new_col"] = **"group2"**
sc.tl.rank_genes_groups(adata, "new_col", groups=(["group1"]), reference = "group2", method='wilcoxon')
sc.pl.rank_genes_groups(adata, groups=['group1'], n_genes=15)
sc.pl.rank_genes_groups_violin(adata, groups=['group1'],n_genes=15)

The script keeps hanging on the line with '& - sign'. or there are errors with invalid syntax. Really not sure what I am doing wrong, I have read the documentation that you suggested. Thank you, Josine

Hi Josine,
can you please always include the complete error message? Otherwise it's hard to tell what the actual problem is.
In the line

adata.obs.loc[is_shared & is_brain), "new_col"] = "group2"

I spot an unmatched bracket is_brain). Maybe that's already the problem?
Best,
Gregor

Hi Gregor,
So yesterday and today no error, just hangs forever (see above). I have restarted the kernel multiple times, but it doesn't seem to work. As you can see I use somewhat different names to describe the groups than I used above, so please don't get confused :) -> It should be pretty straight-forward what indicates the different groups. Thanks for your time, best, Josine

is_tumor correctly is a boolean mask, i.e. a pandas series of booleans
is_shared, however, is an object of class anndata (-> you'll need to get rid of the outer final_data[...])

tbh, I don't know why this is not throwing an error though

Hi Gregor,
This is just a personal mail, so not really intended to be posted on the web:
Can't thank you enough, you helped me a great deal. I just started my PhD here at Yale as a Dutch M.D., and without any prior coding experience it has been quite the challenge to make Python work. But thanks to your clear instructions, I not only booked some great results, but also learned a lot, and became enthusiastic about coding - and learn what it can mean to medical science. So thanks again, and for contributing to my Phd, really. If there is anything I can do in return, let me know.  I’ll definitely make sure the Scirpy package will be named in any further publications.
Best,
Josine
Josephina Jansen, M.D.
Yale School of Medicine | Department of Neurology | Hafler Lab
Combined PhD Program Utrecht University | Clinical and Experimental Neuroscience
<EMAIL_ADDRESS>Phone: + 1 (203) 809 - 4635
On Nov 18, 2020, 2:41 AM -0500, Gregor Sturm<EMAIL_ADDRESS>wrote:

is_tumor correctly is a boolean mask, i.e. a pandas series of booleans
is_shared, however, is an object of class anndata (-> you'll need to get rid of the outer final_data[...])

tbh, I don't know why this is not throwing an error though
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ficbi-lab%2Fscirpy%2Fissues%2F204%23issuecomment-729497685&data=04|01|josephina.jansen%40yale.edu|f58a2da9043b4bd986f808d88b956710|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637412821080440701|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=EddktWY%2BbFgRDu8SDGBuglKBwcqdAtb60BGRH3RNJTY%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAQ355Z7ZG7BMRPWDX5436YTSQN3DTANCNFSM4SRNZV6A&data=04|01|josephina.jansen%40yale.edu|f58a2da9043b4bd986f808d88b956710|dd8cbebb21394df8b4114e3e87abeb5c|0|0|637412821080450693|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|1000&sdata=IqfXw2wFDCoMHZuTVQU39DlNQbihwJhcKzAj%2BGjoNW0%3D&reserved=0.

Hi Josine,
I'm closing this issue for now. Feel free to reopen, or create a new one, if you have further questions.
Best,
Gregor


FRESH DROPLET Unable to clean OpCache

On a fresh droplet and EE installation
root@loophole:/var/www/loophole.io# ee clean --all
Restart : memcached [OK]
Cleaning MemCache
Cleaning NGINX FastCGI cache
Cleaning opcache
Unable to clean OpCache
root@loophole:/var/www/loophole.io#

Plz Check if you have installed admin tools first , if not then install them with ee stack install --admin command. and then you can use ee clean --all command

@harshadyeola that was it! Simple solve. Thank you

Could there be another solve? I have the same problem and ee stack install --admin has been initiated already, so for good measure I ran gain - still the notice persists


Autoscaling

Hi, is it possible to integrate the cluster autoscaler with the machine-controller? Or does the controller provide the autoscaling features?

Hey, thanks for inquiring. What we need is a cluster-autoscaler implementation that supports clusterapi v1alpha1 as backend. @frobware has built one already, althought it isn't merged: https://github.com/kubernetes/autoscaler/pull/1866
Other than that, we'd need to fix https://github.com/kubermatic/machine-controller/issues/564 (althought that doesn't matter if you run on a platform that does have a cloudprovider)

thanks @alvaroaleman

/remove-lifecycle

/remove-lifecycle stale

/remove-lifecycle stale

So, upstream clusterAPI change got merged and released. And machine-controller doesn't need to do anything!
In order to make autoscaling work, MachineDeployments or MachineSets should be annotated like this:
apiVersion: cluster.k8s.io/v1alpha1
kind: MachineDeployment
metadata:
  annotations:
    cluster.k8s.io/cluster-api-autoscaler-node-group-min-size: "1"
    cluster.k8s.io/cluster-api-autoscaler-node-group-max-size: "10"

But such annotations are out of machine-controller scope and should be provided by the user.
See more info at https://github.com/kubernetes/autoscaler/pull/1866

May I ask how the autoscaler annotation could be used in the output.tf kubeone_workers?
From the examples there is a "$var.cluster_name}-pool1" example.

Terraform output is unmarshalled onto
https://github.com/kubermatic/kubeone/blob/8418d9d6c14e38a525618829fc1c36d56a8c3f0a/pkg/apis/kubeone/v1beta2/types.go#L418-L425
so there is annotations field that will be copied to MachineDeployment.ObjectMeta.Annotations
https://github.com/kubermatic/kubeone/blob/8418d9d6c14e38a525618829fc1c36d56a8c3f0a/pkg/apis/kubeone/v1beta2/types.go#L432
so the resulted terraform output would look something like this:
output "kubeone_workers" {
  value = {
    "${var.cluster_name}-pool1" = {
      replicas = var.workers_replicas
      providerSpec = {
        annotations = {
          "cluster.k8s.io/cluster-api-autoscaler-node-group-min-size" = "1"
          "cluster.k8s.io/cluster-api-autoscaler-node-group-max-size" = "10"
        }
...

@kron4eg Thank you so much mate!


Update DCOS Version to RC3.

High-level description
Update version information to 1.11.0-rc3

@mesosphere-mergebot bump-ee

@mesosphere-mergebot label ready for review

Going ahead with applying a ship-it label for the version change so that we can include it in the next train.

@mesosphere-mergebot label

@mesosphere-mergebot label ship-it


Topic

Your Workstation
Windows 10 Laptop, 16 GB RAM, 8 core i7 CPU
What happened?
Include topic
Relevant output
Hi Sir,

Please include external name service.

Not sure what you mean.
This repo is not about doing things on a working cluster. It merely serves to illustrate what is involved to build a highly available cluster from scratch, not using any tools that assist with e.g. creating PKI as kubeadm does for you.


Support WS connection

Implement WebSocket connection support in the client and add a few examples on how to use it to the documentation

WebSocket was supported in the client


[Array.prototype.reduce] TS infers accumulator type as any[]

TypeScript Version:  3.7-Beta, "noImplicitAny" compiler mode
Search Terms:
Array.prototype.reduce, reduce type inference, reduce implicit any
Code
const result = [1].reduce((acc, item) => {
    acc.push(item)
    return acc
}, [])

result[0].toLowerCase()

Expected behavior:

result implicitly has type any[] -> compiler error ("noImplicitAny")
or
result has type unknown[] -> compiler error

Actual behavior:
result has type any[], no errors
Playground Link: http://www.typescriptlang.org/play/?strictNullChecks=false&strictPropertyInitialization=false&ts=3.7-Beta&ssl=6&ssc=24&pln=1&pc=1#code/MYewdgzgLgBATgUwgVwDawLwwNoEYC6AdIgCbLAIAUlAhsMADQwCWUCAtgJQwYB8MAbwBQMUTDrBCAB2QQAFpVYdOIsYijI4YcfSEBfJtnwqhiFOmwAGIlBAAZEAHcEcAMI0IVTkA
Related Issues: https://github.com/microsoft/TypeScript/issues/25454, https://github.com/microsoft/TypeScript/issues/29604
P.S. It seems reasonable to align behavior with other empty array cases, e.g.:
let x = {
    array: [] // Object literal's property 'array' implicitly has an 'any[]' type
}

let x = []
x[0].toLowerCase() // Variable 'x' implicitly has type 'any[]' in some locations where its type cannot be determined

We should try to write a new definition of reduce that produces the desired behavior in all cases


How to bootstrap into an empty directory

If I want to use this flake in an empty directory I cannot since I do not have a cargo.toml file, and no cargo to make it. I can just manually create the file but this seems unnecessary no?

If, in that empty directory, you run nix flake init -t github:accelbread/flakelight-rust it will set up a starting project for you with a Cargo.toml and ready for use with nix develop.
Does that meet your needs?

Added 5e2e3ed28f547e7fd76bc9126251505739586091 which enables using the devShell without a Cargo.toml present.


"what is the capital of new york" is not working

The response is in div.kno-fb-ctx. Can you also add a lookup for that div for the answer? I could fork and add, but looks like you are in the process of creating a version 2 so I figured it would be easier to file an issue instead.

I've just tried that question and I get a valid response:-
New York, Capital , Albany,

Agree. I tried as well and got a valid response today.


[List] Inherit list item icon color from the parent

This is a pull request fixes #26267

[x] I have followed (at least) the PR section of the contributing guide.

@Goodiec please open a new pull request when working on a new issue.

@Goodiec please open a new pull request when working on a new issue.

Ok, will do.

Looking at the visual regressions, it's not working

Looking at the visual regressions, it's not working

Agree, looks like the ListItemIcon is used also purely for setting up the correct layout. With the change all icons have opacity.

Thank you @mnajdova, I'll close this PR


Extend RUN VIEW command to display only metrics

We need to rethink how to show the metrics. At the end it has to be some kind of table like this:
+--------------------------+----------+--------+-------+-------+-------+-------+--------+
|         process          |  metric  |  mean  |  min  |  q1   |  q2   |  q3   |  max   |
+--------------------------+----------+--------+-------+-------+-------+-------+--------+
| RSEQC_READDUPLICATION    | cpuUsage | 219.45 | 153.8 | 173.3 | 189.4 | 250.5 | 330.25 |
| RSEQC_JUNCTIONSATURATION | cpuUsage |  227.1 |  93.2 | 209.2 | 209.4 | 308.6 |  315.1 |
+--------------------------+----------+--------+-------+-------+-------+-------+--------+

So I think that it is better to remove metrics from the VIEW command and move them into a new command.
I think that it will be more useful if we move them into a subcommand like this:
tw runs view -i <run_id> metrics -f <filter_process> --type cpu,mem,vmem,time,reads,writes,cpuUsage,memUsage,timeUsage --columns mean,min,q1,q2,q3,max,minLabel,q1Label,q2Label,q3Label,maxLabel,warnings

-f is optional and by default all the process are shown
--type is optional and the default is cpu,mem,time
--columns is optional and the default is mean,min,q1,q2,q3,max

Let's think about having a * type and columns to prevent "hidden" metrics that can come in future release of nextflow


[AutoPR recoveryservicessiterecovery/resource-manager] [Hub Generated] Review request: Enhanced A2ARecoveryPointDetails with "disks"

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/5307

Build output:

dist/com.microsoft.azure.recoveryservicessiterecovery.v2018_01_10/azure-mgmt-recoveryservices-1.0.0-beta-javadoc.jar
dist/com.microsoft.azure.recoveryservicessiterecovery.v2018_01_10/azure-mgmt-recoveryservices-1.0.0-beta.jar
dist/com.microsoft.azure.recoveryservicessiterecovery.v2018_01_10/azure-mgmt-recoveryservices-1.0.0-beta-sources.jar
dist/com.microsoft.azure.recoveryservicessiterecovery.v2018_01_10/azure-mgmt-recoveryservices-1.0.0-beta.pom

This PR has been merged into https://github.com/Azure/azure-sdk-for-java/pull/3037


kick out partial matches on DID parse

e.g.,
did:bad:char'@='acters:example:1234abcd#4
should fail to parse
Signed-off-by: sklump<EMAIL_ADDRESS>

I wouldn't have expected that a small regex change would break everything lol but I guess that's regex lol. The DID_PATTERN is reused in a couple of places: in the did.py, where DID_PATTERN.fullmatch is used to remove the partial match cases and again in did_url.py to match and remove the first part of the DID url to reuse urlparse from the standard library to parse the rest of the url. Now that you call this out, the DID URL parsing does have partial match issues:
    def test_partial_match_url():
>       assert not DIDUrl.is_valid("did:bad:char'@='acters:example:1234abcd#4")
E       assert not True
E        +  where True = <bound method DIDUrl.is_valid of <class 'pydid.did_url.DIDUrl'>>("did:bad:char'@='acters:example:1234abcd#4")
E        +    where <bound method DIDUrl.is_valid of <class 'pydid.did_url.DIDUrl'>> = DIDUrl.is_valid

#24 merged, addressing the regex fix added with this PR. Closing.


Replace ref-counting in tail call path with epoch logic

The tail call path currently acquires and releases a global lock (_ebpf_object_tracking_list_lock) twice during a tail call:

During the bpf_tail_call helper (to grab a reference on the object)
After the tail call completes (to release the reference on the object).

Given that program and map objects are protected by epoch logic, it should be possible to do this without needing to grab a reference to the object.

Alternatively, we can also use per-object lock, and a "deleting" state, which will remove need to acquire global locks for each acquire / release of ref:
Acquire ref:

Call interlockedIncrement

Release ref:

Acquire object lock
Call InterlockedDecrement
If ref is 0 ==> mark object as "deleting"
Release object lock
Delete object from hash table.

Query hash table:

Acquire table lock and find object using the key
Acquire object lock
If object is "deleting", return NOT FOUND
Else, acquire reference on the object.
Release object lock
Release table lock


Footer Menu on Post hidden when you scroll

Menu

Watch the gif, this occur when you scroll through the comment of a post

@horpeyloaded it is not hidden it is intentional to show it attached on footer only when you scrolling post, after post is done, you likely reading comments and want to reply to comment, instead of post.

Good to know and thanks @feruzm. I wanted to ask that was it a feature or a bug. I forgot 🗳.


add some Pekko libs to README.md

Equivalents of some Akka libs that are listed already.

thanks!


How to changing the framing size of BarcodeView?

Changing the framing rect size in width and height in xml changes the physical size of the view but the image from the camera is scaled meaning the scanning size is still the same

You can specific the size of the framing rect as in this example: https://github.com/SunhoY/zxing-android-embedded/blob/master/sample/src/main/res/layout/custom_barcode_scanner.xml#L11-L12

doesn't work for me. The size of surface view still large than barcode view size.

There are some fixes for this in v3.2.0.
New instructions here:
EMBEDDING.md

Is version 3.2.0 available now? Couldn't include it to my project.
2016-02-06 17:56 GMT+03:00 Ralf Kistner<EMAIL_ADDRESS>
Closed #115
https://github.com/journeyapps/zxing-android-embedded/issues/115.
—
Reply to this email directly or view it on GitHub
https://github.com/journeyapps/zxing-android-embedded/issues/115#event-541741272
.

Version 3.1.0 not available now too :(
2016-02-08 10:52 GMT+03:00 Raman Branavitski<EMAIL_ADDRESS>
Is version 3.2.0 available now? Couldn't include it to my project.
2016-02-06 17:56 GMT+03:00 Ralf Kistner<EMAIL_ADDRESS>
Closed #115
https://github.com/journeyapps/zxing-android-embedded/issues/115.
—
Reply to this email directly or view it on GitHub
https://github.com/journeyapps/zxing-android-embedded/issues/115#event-541741272
.

Yes, it's released. What is the specific issue that you get? Are you using the jcenter() repo?

I am sorry, it's android studio bug. Write that can not resolve any class
from your library, but project build successfully. Sorry.
2016-02-08 11:20 GMT+03:00 Ralf Kistner<EMAIL_ADDRESS>
Yes, it's released. What is the specific issue that you get? Are you using
the jcenter() repo?
—
Reply to this email directly or view it on GitHub
https://github.com/journeyapps/zxing-android-embedded/issues/115#issuecomment-181251550
.


Source maps broken / don't load in firefox

Node version: 0.10.36
Firefox version: 38.0.5
Steps to reproduce

Clone source and do npm install
Do npm run test
Do python3 -m http.server
Open Firefox and open the dev console (Debugger tab)
Got to http://<IP_ADDRESS>:8000/test/sourcemap_tester.html

In the inspector should see:
Error loading source:
loadSourceError

Edit: this seems to happen with sourceMapStyle embed or inline but not link

Update: this bug appears to fixed in the dev build of Firefox. So maybe it's probably a bug in Firefox opposed to a bug in how we generate sourcemaps.

So in ~12 weeks should see this fix in release version of Firefox


@storybook/addon-comments v3.0.0 falls with error

VM9775 manager.bundle.js:94268 Uncaught TypeError: Cannot read property 'persister' of null
    at Container.init (VM9775 manager.bundle.js:94268)
    at Container.componentDidMount (VM9775 manager.bundle.js:94243)
    at VM7843 manager.bundle.js:28100
    at measureLifeCyclePerf (VM7843 manager.bundle.js:27910)
    at VM7843 manager.bundle.js:28099
    at CallbackQueue.notifyAll (VM7843 manager.bundle.js:11842)
    at ReactReconcileTransaction.close (VM7843 manager.bundle.js:32993)
    at ReactReconcileTransaction.closeAll (VM7843 manager.bundle.js:5946)
    at ReactReconcileTransaction.perform (VM7843 manager.bundle.js:5893)
    at batchedMountComponentIntoNode (VM7843 manager.bundle.js:12750)

also
VM9775 manager.bundle.js:93970 Uncaught TypeError: Cannot read property 'persister' of null
    at DataStore._loadUsers (VM9775 manager.bundle.js:93970)
    at VM9775 manager.bundle.js:93917

Hi @dangreen, thanks for reporting this. I just confirmed this with both the test-cra app and a vanilla CRA app.
The issue is the lack of a database package. See https://github.com/storybooks/storybook/pull/1145.
I'm not totally sure how it worked before, or how to best move forward @ndelangen, but at the least we should indicate this problem on the comments addon for now.

@ndelangen -- what are your thoughts here?

@tmeasday @ndelangen let's get rid of comments in 3.2! 👍

@ndelangen confirmed to me he agrees we should remove it! If anyone objects, now would be a good time :)

Remove it, we will rebuild something like it when the time comes.

I'm interested in this comments addon. Was the consensus to not work on this for the time being? What needs to be done?

@jakiestfu I don't think we want to write custom connectors for all sorts of databases or chat apis... I'm not 100% sure on this, but it's how I feel about it now.
Of course I wouldn't stop anyone from writing such a thing, though we'd have to argue to add it to the core repo or not.
I'm not sure there's consensus about it, I'm personally a little divided on the issue, to be honest.
I have mentioned in the past that I think a good minimal viable solution would be to embed a system like https://disqus.com/ into an panel, what are your thoughts on that?


Add Exceptions Support

It would be nice to check how jeo can transform bytecode with try-catch, try-catch-with-resources and Exceptions in general. If jeo can't handle exceptions, we should definetly implement this.

@volodya-lombrozo 2 puzzles #324, #325 are still not solved.

@volodya-lombrozo the puzzle #325 is still not solved; solved: #324.

@volodya-lombrozo 2 puzzles #325, #329 are still not solved; solved: #324, #330.

@volodya-lombrozo 2 puzzles #325, #335 are still not solved; solved: #324, #329, #330, #336.

@volodya-lombrozo the puzzle #325 is still not solved; solved: #324, #329, #330, #335, #336.


giomodule.cache conflict / collision

Basic info

Kernel: 3.18.25
System: 15.09.git.e83aed4 (Dingo)
Nix version: nix-env (Nix) 1.10
Nixpkgs version: "16.03.git.c6fcf44"

Describe your issue here
There is the following collision when I install my profile:
collision between `/nix/store/vfqb0prz2ilfksl65k1w3wyp8bsa7zxl-gvfs-1.22.2/lib/gio/modules/giomodule.cache' and `/nix/store/qykzxrzhaj9g5j9nvkcyraqnmr4fdvbp-dconf-0.24.0/lib/gio/modules/giomodule.cache'

there seem to be these giomodule.cache files created for gnome3 packages. I'm
not sure how this should be handled. Should we use similar solution as for the
hicolor/icon-theme.cache issue (See #10370) and delete these cache files?
Expected result
No collision.
Actual result
Described collision.
Steps to reproduce
Install both dconf and any package that depends on gvfs (I believe
gnome3.file-roller in my case) in the same profile.

@lethalman: do you know if giomodule.cache is of any use in our model?
I think we always compose by setting $GIO_EXTRA_MODULES,  either by hand or by wrapGAppsHook.

@vcunat that I know, it has never been useful. Most of apps already use stuff like glib networking and dconf module, without the need of any cache.

So it should be OK to add a hook to remove those caches, right?

@vcunat it should, but I haven't tried.

triage: still a problem?


[BUG] regenerate_dataset() method yielding RuntimeError: cuDF failure at: ... /dask_sink.cpp:37 when using gcs protocol

Describe the bug
Using regenerate_dataset() method on a merlin.io.dataset.Dataset results in error RuntimeError: cuDF failure at: /workspace/.conda-bld/work/cpp/src/io/utilities/data_sink.cpp:37: Cannot open output file when using gcs protocol.
Full traceback is:
/opt/conda/miniconda3/lib/python3.8/site-packages/merlin/io/parquet.py:751: UserWarning: General-metadata information not detected! Please pass lists for `cats`, `conts`, and `labels` asarguments to `regenerate_dataset` to ensure a complete and correct _metadata.json file.
  warnings.warn(
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_27088/1285145988.py in <cell line: 3>()
      1 import gcsfs
      2
----> 3 nvt_test_ds_regen = nvt_ds.regenerate_dataset(
      4     output_path=os.path.join("gs://", BUCKET, 'tmp/data_dir/data_regen/'),
      5     part_size=128_000_000,

/opt/conda/miniconda3/lib/python3.8/site-packages/merlin/io/dataset.py in regenerate_dataset(self, output_path, columns, output_format, compute, **kwargs)
   1112         result = ParquetDatasetEngine.regenerate_dataset(self, output_path, columns=None, **kwargs)
   1113         if compute:
-> 1114             return result.compute()
   1115         else:
   1116             return result

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/base.py in compute(self, **kwargs)
    313         dask.base.compute
    314         """
--> 315         (result,) = compute(self, traverse=False, **kwargs)
    316         return result
    317

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/base.py in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    598         postcomputes.append(x.__dask_postcompute__())
    599
--> 600     results = schedule(dsk, keys, **kwargs)
    601     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    602

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/threaded.py in get(dsk, keys, cache, num_workers, pool, **kwargs)
     87             pool = MultiprocessingPoolExecutor(pool)
     88
---> 89     results = get_async(
     90         pool.submit,
     91         pool._max_workers,

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/local.py in get_async(submit, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, chunksize, **kwargs)
    509                             _execute_task(task, data)  # Re-execute locally
    510                         else:
--> 511                             raise_exception(exc, tb)
    512                     res, worker_id = loads(res_info)
    513                     state["cache"][key] = res

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/local.py in reraise(exc, tb)
    317     if exc.__traceback__ is not tb:
    318         raise exc.with_traceback(tb)
--> 319     raise exc
    320
    321

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
    222     try:
    223         task, data = loads(task_info)
--> 224         result = _execute_task(task, data)
    225         id = get_id()
    226         result = dumps((result, id))

/opt/conda/miniconda3/lib/python3.8/site-packages/dask/core.py in _execute_task(arg, cache, dsk)
    117         # temporaries by their reference count and can execute certain
    118         # operations in-place.
--> 119         return func(*(_execute_task(a, cache) for a in args))
    120     elif not ishashable(arg):
    121         return arg

/opt/conda/miniconda3/lib/python3.8/site-packages/merlin/io/parquet.py in _write_data(data_list, output_path, fs, fn)
    922     for data in data_list:
    923         rows += len(data)
--> 924         writer.write_table(cudf.from_pandas(data))
    925
    926     # Return metadata and row-count in dict

parquet.pyx in cudf._lib.parquet.ParquetWriter.write_table()

parquet.pyx in cudf._lib.parquet.ParquetWriter._initialize_chunked_state()

RuntimeError: cuDF failure at: /workspace/.conda-bld/work/cpp/src/io/utilities/data_sink.cpp:37: Cannot open output file

Steps/Code to reproduce bug
import cudf
import nvtabular as nvt

df = cudf.DataFrame({'a': list(range(20)),
                     'b': list(reversed(range(20))),
                     'c': list(range(20))
                    })

nvt_ds = nvt.Dataset(
        path_or_source=df,
        cpu=False,
)

nvt_ds_regen = nvt_ds.regenerate_dataset(
    output_path=os.path.join("gs://", BUCKET, 'tmp/data_dir/data_regen/'),
    part_size=128_000_000,
    file_size=128_000_000 * 10,
    compute=True,
)

Expected behavior
Expect to see parquet files in specified GCS bucket such as:
gs://{BUCKET}/tmp/data_dir/data_regen/part.0.parquet
gs://{BUCKET}/tmp/data_dir/data_regen/part.1.parquet
...

Environment details (please complete the following information):

Environment location: Google Cloud Platform Single Node VM Dataproc Cluster

Ubuntu 18
Python 3.8
NVIDIA Tesla A100 GPU
--scopes cloud-platform,storage-full,storage-rw

Versions of relevant libraries:
fsspec=='2022.5.0'
gcsfs=='2022.5.0'

CUDA_VERSION=11.7
CUDF_VERSION=22.10.0
SPARK_RAPIDS_VERSION=22.10.0
DASK_RAPIDS_VERSION=22.10
DASK_VERSION=2022.9.2
DASK_CUDA_VERSION=22.10.00
NVTABULAR_VERSION=1.5.0
MERLIN_CORE_VERSION=0.8.0
CUDA_PYTHON_VERSION=11.7.0
CUPY_VERSION=9.6.0

RAPIDS_RUNTIME=DASK
DASK_RUNTIME=standalone
INSTALL_GPU_AGENT=true
DASK_CUDA_WORKER_ON_MASTER=true

GPU_TYPE=nvidia-tesla-a100
N_GPUS=1
MASTER_MACHINE_TYPE=a2-highgpu-1g

Method of NVTabular install: [conda, Docker, or from source]

 mamba install -y --no-channel-priority -c 'rapidsai' -c 'nvidia' -c 'numba' -c 'conda-forge' \
   "nvtabular=${NVTABULAR_VERSION}" \
   "merlin-core=${MERLIN_CORE_VERSION}" \
   "dask-cuda=${DEFAULT_DASK_RAPIDS_VERSION}" \
   "cuda-python=${CUDA_PYTHON_VERSION}" \
   "cupy=${CUPY_VERSION}" \
   "cudatoolkit=${CUDA_VERSION}" \
   "rapids=${RAPIDS_VERSION}" \
   "cuml=${CUDF_VERSION}" \
   "cudf=${CUDF_VERSION}" \
   "cucim=${CUDF_VERSION}" \
   "ucx-proc=*=gpu" \
   "ucx-py" \
   "rmm" \
   "betterproto" \
   "grpclib" \
   "jupyterlab-nvdashboard"

Additional context
Looking at the source code here I was trying to see if this was a permissions issue with the fsspec GCSFileSystem but I am able to see that my credentials, access, and tokens appear fine in the step:
from fsspec.core import get_fs_token_paths

storage_options = {
        'anon': True,
        'use_ssl': False,
    }

fs, fs_fs_token, paths = get_fs_token_paths(
    urlpath=os.path.join("gs://", BUCKET, 'tmp/data_dir/data_regen/'),
    mode="wb",
    storage_options=storage_options)

>> fs.credentials.credentials.scopes
['https://www.googleapis.com/auth/devstorage.full_control']

and I can also see that a token is generated and that it recognizes the project ID. I have no problem writing to this bucket using other frameworks (e.g. using PySpark, Pandas, pyarrow, etc.).
I dug around and this issue stuck out to me (GCSFileSystem hanging when called from multiple processes), but I am not sure if this is the right direction.

Thanks for raising an issue @njelmert !
I don't think to_parquet/regenerate_dataset were designed to handle remote storage yet, but I could be wrong. Either way, we should have the necessary tools in cudf/fsspec to do this now, so I'll try to figure out what needs to change.

Thank you, @rjzamora ! I hadn't considered that not being a current capability so that's good to know. For now I have a work-around which temporarily dumps the parquet files to disk and gsutils them to remote storage. Looking forward to the adaptation.


Add tcmalloc for amd64 platform

Depends on https://github.com/rancher/tgt/pull/8 and https://github.com/rancher/liblonghorn/pull/16
Longhorn 4136
Signed-off-by: Derek Su<EMAIL_ADDRESS>

BTW, is this verifiable? Could we understand any obvious performance difference after this introduction? If yes, that will be good.

@derekbit https://github.com/rancher/tgt/pull/8 and https://github.com/rancher/liblonghorn/pull/16 all merged. Do you think this's ready for review?

@derekbit need to backport to 1.3.1. Thanks.


dockerignore is no longer needed.

This is handled by the docker plugin resource section

rebased and merged as https://github.com/spring-cloud/spring-cloud-data/commit/a8533228a4dceac204d7f8d5f74f6d4e9e197293


How to setup the environment?

Should I follow the README.md in the file MKDNet.rar ?
Or I should download the latest code from pytracking and use the model you provided?

You can use the official settings of pytracking for the code environment, and use our code.
------------------ 原始邮件 ------------------
发件人:                                                                                "ZYB0726/MKDNet"                                                                                @.>;
发送时间: 2023年2月13日(星期一) 晚上6:59
@.>;
@.***>;
主题: [ZYB0726/MKDNet] How to setup the environment? (Issue #8)
Should I follow the README.md in the file MKDNet.rar ?
Or I should download the latest code from pytracking and use the model you provided?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Use Debug Focus + Show Poly Mesh shows strange results

I made a short video to illustrate the behaviour, not sure if intended but the appearance of the navmesh reads like a bug.
http://files.facepunch.com/billowe/2016/November/02/f359d/2016-11-02_14-47-17.mp4

The debug vis. was pulled from 5.6 - We're reworking this now.


[fix][broker]Add origin signature for compatible.

Fixes #xyz

Master Issue: #xyz

Motivation
Some users may the following method to create resources.
public static MetadataStoreExtended createLocalMetadataStore(String serverUrls, int sessionTimeoutMs)

In #19156, we change the signature, which may introduce the compatible problem. So here we maintain the origin signature.

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Some users may the following method to create resources.

Do you have a real-world report? I'm wondering how an internal method (broker is not exported to user) gets used.

Yes, it's a little strange. It should only be used on the pulsar broker side.
In the pulsar function mesh, it uses PulsarResources to create a meta store to debug. So the function mesh compiles error using the new release.
At now, the pulsar function mesh removes the usage.


Single DLL

Hi,
I am trying to create for my own use, a single Dll which has embedded a copy of the eng.traineddata, leptonica-1.80.0.dll and also tesseract41.dll.
The problem I am having is that I cannot locate the code that loads the traineddata file.
I primarily code in vb.net however I can understand and code in C# as well.
Please can you give me a few pointers on where to look.
Thanks,
Regards,
Paul.

I think it's happening in tesseract41.dll itself.
https://github.com/tesseract-ocr/tesseract/blob/4.1/src/api/baseapi.cpp#L346

Hi Dmitry,
Thanks for the quick reply. I suspected that would be the case.
That makes my whole project a lot more complicated.
Regards,
Paul.


AnalysisException: Unable to infer schema for CSV. It must be specified manually.

Dear all,
the code upsertDataDF = (spark   .read   .option("header", "true")   .csv(inputPath) ) upsertDataDF.createOrReplaceTempView("customer_data_to_upsert")
runs into

AnalysisException: Unable to infer schema for CSV. It must be specified manually

How can this be fixed?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3fc98932-2ac8-45b7-3609-386626bc41f4
Version Independent ID: ea9fe6d2-a1ab-9b43-d06c-c1ad7edd7bcf
Content: Tutorial: Use Azure Data Lake Storage Gen2 events to update a Databricks Delta table
Content Source: articles/storage/blobs/data-lake-storage-events.md
Service: storage
Sub-service: data-lake-storage-gen2
GitHub Login: @normesta
Microsoft Alias: normesta

Hello @AlexanderTodorovic  and thank you for your inquiry.  We will review the issue and get back to you.  Could you please specify what service you are using?  (DataBricks, HDInsight, other)

@AlexanderTodorovic  I have followed the tutorial up until step Create a Job.  I have not run into the error message you reported.
The error message sounds like what would happen if steps 3 and 4 under 'Create and populate a Databricks Delta table' did not execute successfully.
Could you please try those steps again?  Step 3 could fail if the 'adlsPath' from Step 1 doesn't correspond to your storage account.
adlsPath =<EMAIL_ADDRESS>adlsPath =<EMAIL_ADDRESS>

@MartinJaffer-MSFT The delta table is created successfully. The folder delta-tables/customers was created and the ...snappy.parquet file was also created. However the documentation says this cell shouldn'be run. Because later in the tutorial the function passes a value to the source_file widget and I think it'll be fine. Thank you!


Add LICENSE.md file

In the License & Copyright section we state, "See the included LICENSE.md file for more details." Though that file does not actually exist.

because apparently I don't know how to use git add


FBX Importer fails to import smooth shaded objects

I created a simple smooth-shaded UV sphere in Blender (2.90 and 2.91) and exported it as fbx file.

After importing the fbx file into Godot, it looks flat shaded.

Importing the same fbx file back into Blender works just fine. Triangulating the sphere before exporting had no effect.
Godot versions:
v3.2.3 mono win64
v3.3 mono win64
I attached the sphere model. If there's anything else you need, let me know.
sphere.zip

Duplicate of https://github.com/godotengine/godot/issues/46851.


Feature/include/session on registration

@littleforest  should be good-to-go once the specs clear.


Implement symbol table and Symbol core class in Rust

This PR implements the ruby symbol table using this project's intaglio
crate, exposes access to the table by implementing all mruby sym
MRB_API functions, and adding a Symbol class implementation that
boxes into a normal MRB_TT_SYMBOL mrb_value.
artichoke-backend does a custom integration of intaglio that makes
the following observable changes to its API:

Returned symbol identifiers are one-indexed.
All interned content is inspected and potentialy modified to be
NUL-terminated.

The Symbol class is implemented with a newtype around intaglio's
u32 symbol id newtype. This PR differentiates between heap allocated
data and Copy immediates when implementing BoxUnboxVmValue which
allows Symbol to be unboxed without a heap allocation or Box. This
also enables artichoke-backend to box Symbol values into the
mrb_value tagged union as an MRB_TT_SYMBOL. Because of this,
Symbol does not require a free implementation.
These changes allow many of the sym APIs in mruby-sys/ext to be
removed. The only one that remains does the fiddly union handling to
properly setup the tagged union in an mrb_value for a Symbol.
Interpreter initialization has been reworked to allocate the State
before calling mrb_open. To facilitate this, the parser is lazily
initialized (it is not needed during interpreter boot). This change
enables injecting the state, and symbol table, in at mrb boot via the
alloc_ud parameter. ffi::from_user_data gets support for unpacking
the State from this field on the mrb_state.
TryConvertMut impl for &str and &[u8] have changed to no longer
convert MRB_TT_SYMBOL due to lifetime issues. To work around this,
converters relying on this behavior can return a copy of the symbol
after unboxing it so callers can resolve the byte content themselves. An
example of this can be seen in the CaptureExtract converter in
MatchData#offset.
Additional converters for usize, isize, and Cow<'_, OsStr> have
been added. Cow converters have been optimized to send owned content
and borrowed content through separate converter paths.
To implement the MRB_API for symbol, symbol.c in
mruby/src/symbol.c has been #ifdef'd out and replaced with
#[no_mangle] functions in extn/core/symbol/ffi.rs. These functions
delegate directly to the Intern impl on Artichoke.
Symbol core class implements all APIs, including
Symbol::all_symbols. Symbol#inspect does not print the byte
contents. See GH-725.
There is one nasty bit of unsafe in
Value::implicitly_convert_to_string which uses mem::transmute to
modify the lifetime of the returned interned bytes from the Artichoke
to the Value. This is safe because Values must live shorter than the
attached interpreter. This is not enforced since Values are Copy.
Fixup or remove tests that no longer pass with Rust symbol table
The sys module is no longer standalone in an Artichoke configuration
which defines ARTICHOKE during the mruby compile. mrb_open without
an Artichoke State will segfault.

The Symbol to String converter was explicitly removed because of lifetime
issues.
FFI tests were removed because the mruby C code no longer is memory safe
without the Artichoke internals.
Interpreter debug uses crate::interpreter to ensure a fully formed interpreter.

This PR does not implement Symbol#inspect (see GH-725).
Fixes GH-219.
Fixes GH-232.
Fixes GH-71.
Relates to GH-725.
Relates to GH-442.
Relates to GH-670.
Performance
I hypothesized that a Rust implementation of the Ruby symbol table would outperform the mruby C implementation.
Methodology

cargo build --workspace --release this branch @ b313130.
Remove env suite from spec-runner/enforced-specs.yaml.
hyperfine --warmup 10 --runs 150 --export-json artichoke-symbol.json './target/release/spec-runner spec-runner/enforced-specs.yaml'
cargo build --workspace --release master @ 079545095e0167f270343bda6e0923880edac072.
Remove env suite from spec-runner/enforced-specs.yaml.
hyperfine --warmup 10 --runs 150 --export-json mruby-symbol.json './target/release/spec-runner spec-runner/enforced-specs.yaml'
Import results to Google Sheets.
Run T-test, mean, std deviation, and plot.

Data

Raw timing data in ms

Artichoke (ms)
mruby (ms)

597.2205635
604.303217

629.5361075
623.712414

604.9394955
647.971962

586.7552945
610.772963

627.3469175
620.460868

616.1491715
645.073512

599.5227285
616.094889

622.0094245
620.742602

641.2816745
650.695572

597.9991115
605.878477

615.2613245
611.412492

631.4622755
626.27108

606.1991215
624.118776

592.3681605
610.107989

639.9897705
634.74821

603.9475005
625.142694

595.9535745
602.812126

627.6655995
623.264183

617.8005215
647.920081

593.7965855
607.709305

627.9419205
613.95537

625.9158905
642.562777

589.9231225
622.645392

620.9663415
610.032023

630.3042005
644.275219

605.3502925
625.274442

598.0138295
615.854985

628.3667935
631.41159

606.0235115
610.608538

588.8844705
615.603944

624.6371195
639.112318

627.3905615
628.685391

591.5320545
609.276239

619.7767955
627.220674

627.4643875
631.064037

594.4313035
605.547966

621.9235125
616.827508

630.4176205
652.635632

610.4096145
610.342164

601.3951135
634.532189

654.5323945
662.342494

656.5277445
614.421027

613.3829005
629.245418

631.9478735
642.707545

605.2023855
611.719904

585.8741985
613.805377

628.9276245
634.225417

618.6148605
625.177623

583.8332855
624.651907

618.2428475
617.367099

627.0851635
634.407877

589.0106345
612.16871

602.9961625
648.413782

633.0532305
660.398956

597.5116035
632.935069

600.9808075
661.362845

628.7675915
676.030901

623.1917855
623.765519

603.2989205
632.752569

626.4985655
650.711758

618.1326835
626.500555

591.3107785
615.589353

618.9355355
650.213384

628.4193495
611.590303

592.2121705
620.336501

601.5945955
637.913686

625.7460005
613.681537

595.6012085
617.504238

593.9040875
629.659926

625.2153435
620.275111

615.0970405
614.387391

590.4103225
619.392236

634.4012805
637.195177

627.0388915
602.664473

603.5328615
638.775679

622.5455375
652.524324

632.6561025
605.059581

591.3061605
616.989837

612.9746015
652.836655

632.0834575
607.554985

599.6511245
620.227127

593.9433015
647.031406

626.9795815
610.774878

613.8628365
624.931741

592.0589635
633.56876

628.7570665
626.145803

618.4114455
627.532993

590.1994395
650.799487

615.2835735
641.027373

652.1901705
606.783847

603.5293235
615.254325

622.6593755
635.342962

636.0303665
600.291016

599.5743705
622.717355

594.0576705
654.099055

641.9526575
611.113318

611.2171975
614.876932

599.9072325
652.677945

623.2370875
606.081677

620.5148925
611.978696

596.2324865
653.008914

616.2639695
612.009675

623.9204155
614.367374

592.2170815
622.090336

603.7624425
626.559155

624.4932655
616.95575

610.2013165
616.063389

596.6785335
632.312226

632.3468315
601.291288

608.2277035
615.973823

589.7339745
659.371702

625.6304445
607.244582

611.6558835
614.127218

586.6452095
648.443825

613.9579745
611.784373

631.5959675
616.770265

586.7867155
639.220366

606.0064625
614.720623

626.0340655
600.590329

593.7686195
625.145722

587.9860335
632.116528

624.6976445
613.916684

616.4648685
621.041128

598.5926085
634.206706

621.1017585
600.926155

626.6147575
623.121052

589.6113025
650.029962

610.0497215
617.786667

621.9657815
610.081286

599.8249365
648.637645

598.6564415
615.028838

629.9283095
614.715909

597.0717415
645.722051

592.4666045
621.11946

628.1367105
623.947302

634.7944855
663.455671

597.8736585
632.634291

638.4900365
635.271509

650.7366665
628.377371

598.3698085
623.85582

627.0285465
624.074192

631.5610955
621.944786

586.1461695
640.257295

604.7747625
604.321224

628.9620305
615.658828

604.1346295
653.602009

588.8003375
601.479171

623.8540565
619.645184

613.4204905
650.439317

590.0024735
617.444132

Results
The T-test for these two populations gives a p-value of 0, which indicates that the Rust symbol table has significantly different performance characteristics than the mruby symbol table.
Artichoke with the intaglio-based symbol table completes Artichoke's ruby/spec suite in 97.88% of the time it takes the mruby symbol table to complete the same suite.
Summary

Statistic
Value

T-test
0

Artichoke mean (ms)
612.8211165

Artichoke std dev
16.94718139

mruby mean (ms)
626.0806956

mruby std dev
16.27984529

Artichoke

mruby

This PR was discussed on Reddit and Hacker News.