Enhance TypeScript documentation regarding deployment

Hello CAP Documentation Team,
looking at the documentation page Using TypeScript the part for local development is nicely covered. But when I use cds add mta I don't see the needed compile step being added. Unfortunately the sample repository:
https://github.com/SAP-archive/btp-full-stack-typescript-app
was archived and the branch:
https://github.com/SAP-samples/cap-sflight/tree/typescript
seems not being updated either. I would suggest to add the deployment best practice for CAP TypeScript directly in the documentation.
Best Regards
Gregor

Hi Gregor,
thank you for pointing this out. I'll be working on this and let you know when this is done.
Thanks,
René

Hi Gregor,
half a year ago, @sebastianesch provided the MTA deployment part: https://cap.cloud.sap/docs/tools/cds-typer#integrate-into-your-multitarget-application
When discussing and researching, the question on the use case arose with our colleagues. Having that compile step added to the deployment means that you want to use cds-ts during runtime? If so, would you share the reasoning for that? Maybe @sebastianesch also wants to chime in and share about possible reasons/use cases.
It's especially interesting because we have an explicit note in our docs that states to not use cds-ts productively: https://cap.cloud.sap/docs/node.js/typescript#cds-ts
Looking forward to your thoughts on that.
All the best,
René

Hi René,
I don't want to use cds-ts during runtime. I expect the CAP Documentation to provide the best practice about the build setup for a CAP Applicaiton that uses TypeScript and not just CDS Typer as mentioned in your first paragraph.
In the archived btp-full-stack-typescript-app example it's done in mta.yaml#L16 with npm run build:cf that is executing this scripts from the package.json:
    "build:cf": "npm run build:cds && npm run cleanup:ts && npm run build:ts",
    "build:ts": "tsc",
    "build:cds": "cds build --production",
    "cleanup:ts": "npx rimraf gen/srv/srv/**/*.ts",

Beside this also the tsconfig.json must be provided.
Hope that clarifies my request.
Best Regards
Gregor

Hi Gregor,
thank you for bringing this discrepancy to our attention!
You are right that the current state of the documentation leaves much to be desired and requires the user to figure out a lot by themselves.
We have started investigating how we can accommodate your use case with either tooling from our side or improved documentation. This might take some time as especially the (preferred) tooling approach poses some challenges we have to work out first. But we will get back to you asap.
 I ask, because i have a problem with @impl after upgrading to CAP8. in local dev (cds-ts) on startup it found automaticly the custom implementation, but after ugrading to CAP8 it didn't work in the BTP. we have to add @impl in the cds-file which point to the js-file (one ist TestService.cds, the other is TestService.ts, so we habe to add @impl: TestService.js) This is the workaround to get it running in the BTP, but then the cds-ts didn't start anymore, because while developing there is no TestService.js, only a TestService.ts.


Update version: GyDi.ClashVerge version 1.3.6

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: Open in CodeFlow

Manifest Error: Duplicate installer entry found.


[variables] input-bg and input-color conflict

You must use both $body-bg and $body-color values or $white and $black as default, because now we have unreadable text after changing $body-bg and $body-color values between themselves.
Current values:
$input-bg:                              $white !default; // hardcoded by default
$input-disabled-bg:                     $gray-200 !default;
$input-disabled-border-color:           null !default;

$input-color:                           $body-color !default; // cause unreadable text if $body-color = #FFFFFF
$input-border-color:                    $gray-400 !default;
$input-border-width:                    $input-btn-border-width !default;
$input-box-shadow:                      $box-shadow-inset !default;

If you're overriding some variables, you may have to override additional ones. Not sure I see another way around that at this time.

I see next ways. If it's unavailable for some reasons, i'm sorry.
Way 1
$input-bg:                              $white !default;
$input-color:                           $black !default;

Way 2
$input-bg:                              $body-bg !default;
$input-color:                           $body-color !default;

Ah my bad, if our input bg is set to that we can definitely update it.


Add BaseLoader

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (7e43221) to head (3d7aa22).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #20   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           18        19    +1
  Lines          312       354   +42
  Branches        31        32    +1
=========================================
+ Hits           312       354   +42

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Develop

bugfixes & optimizations galore!

Jenkins now uses a unique NCBI key to prevent clashes

Adding a mutex lock around layouts.p is I think a slightly better solution, but both work :).

Yes, that would a future proof solution! I'll add it to the issues for later integration.

Would be nice if there is some mutex lock that both bash and python understand. That way we could actually have multiple downloads at the same time (id2sra).


[Bug]: after WAU upgrade the "default_excluded_apps" file was overided to default enteries

The problem
It was happened twice, after upgrade to the after 1.21.5 and the second time 1.21.6. The "default_excluded_apps" have been reset to the default values or overrides by them. So that was an annoying to restore form the backup of excluding list after every app upgrade.
What version of WAU has the issue?
1.21.6
What version of Windows are you using (ex. Windows 11 22H2)?
Windows 11 23H2
What version of winget are you using?
v1.8.1911
Log information
#################################################################
#	9/19/2024 - CHECK FOR APP UPDATES (System context - Connected user)
#################################################################
06:55:59 - Notification Level: Full. Notification Language: English
06:55:59 - Checking internet connection...
06:55:59 - Connected !
06:55:59 - Winget Version: v1.8.1911
06:55:59 - WAU current version: 1.21.6
06:55:59 - WAU AutoUpdate is Enabled.
06:55:59 - WAU is up to date.
06:55:59 - WAU uses Black List config
06:56:00 - -> Successsfully loaded default excluded apps list.

Additional information
No response

The default is the default. Normal behavior.
If you want to use a blacklist, you have to bring your own excluded_apps.txt file.

Hi Romanitho,
That what I exactly did, and its worked well for Me until last updates. But I just want to clarify, maybe something changed in the way how the app work, because the folder directory was changed? I mean, if I'll put the excluded_apps.txt into the "C:\ProgramData\Winget-AutoUpdate\config" folder, it should be work. and after next update of app it will be work properly?
Best Regards,

Nope. Nothing has changed. We just crated a default file to config folder, but your list should remain in the root directory.

Thanks


early error out if given ip is already allocated, closes #47

you are amazing! i'm going to close my pr over this!


Book information, crop the text if too long

Context: from the Feedbooks OPDS feed, load "The Great Gatsby" ("Free books", "Most popular"). Open the book information modal window.
The description of this book is very long and a disgraceful scrollbar appear immediately, whatever the size of the window.
A solution is to display 1000 characters max, and add a "Read more" button. If clicked, the remaining text will appear.
Re. accessibility: this button is not shown to the user, as the whole text can be accessed by the screen reader.

Related issue: https://github.com/readium/readium-desktop/issues/370

fixed in bd0a28e


integration tests fail to setup overlay within container launch

During the impl of #9 I ran into issues trying to create a overlay mount from withing another container, which is part of the unit test harness.
This https://github.com/paritytech/sccache/blob/bernhard-podman/src/bin/sccache-dist/build.rs#L273-L307 piece of code errors out with the following error:
(shortened uuids to d9629, added newlines for readbility)
 WARN 2020-11-17T09:31:07Z: sccache::dist::http::server: Res 2 error: run build failed,
caused by: Compilation execution failed,
caused by: Failed to mount overlay FS:
overlayfs "/sccache-bits/build-dir/toolchains/d9629",
upperdir="/sccache-bits/build-dir/builds/d9629-1/upper",
workdir="/sccache-bits/build-dir/builds/d9629-1/work" -> "/sccache-bits/build-dir/builds/d9629-1/target":
Operation not permitted (os error 1) (
"/sccache-bits/build-dir/toolchains/d9629": exists,
upperdir: exists,
workdir: exists,
same-fs,
target: exists,
mapped-root)

To reproduce:
cargo t --features dist-tests test_dist_basic -- --nocapture

in branch bernhard-podman.
Context
The outer container is a rootless podman container.
podman has configurable backends, overlay, vfs, btrfs - the first and last were attempted without any effect.
Adding --privileged or --add-cap CAP_SYS_ADMIN were also attempted for either backend without effect.
Relevant code: https://github.com/paritytech/sccache/blob/bernhard-podman/tests/harness/mod.rs#L354-L387

I see you want to run podman within unprivileged podman. And this is something is rather unreal with gitlab right now.
To start with, there's no clean way to implement the podman runner, can provide you with some details.
Second point is, if there will be a custom runner, it will be just one runner, dedicated to a single project, which is maaaybe a bit of an overkill. Especially if there are another options.
Among another options - gitlab is capable of running dind or the second container as service
Both are are GitLab/Docker hacks™, so should work

for the reference:
https://github.com/containers/podman/issues/3917
https://github.com/containers/podman/issues/4131

I see no lowerdir option speficied for overlayfs.
Regarding upperdir and workdir they looks like on same fs, so it's good.
As I understood from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e9be9d5e76e34872f0c37d72e25bc27fe9e2c54c , without lowerdir , whole overlayfs does not make sense?

for the reference:
containers/podman#3917
obsoleted by the second one :)
containers/podman#4131
and this one says it works for podman inside docker.

docker run --rm -ti \
--security-opt seccomp=unconfined \
--security-opt label=disable \
--cap-add SYS_ADMIN \
--cap-add SYS_RESOURCE \
--env STORAGE_DRIVER=vfs \
quay.io/podman/stable sh -c "podman run hello-world"

What we need is bubblewrap inside podman for step 1, and the make sure that works in the above as well (if we want to go that route).

I see no lowerdir option speficied for overlayfs.
Regarding upperdir and workdir they looks like on same fs, so it's good.
As I understood from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e9be9d5e76e34872f0c37d72e25bc27fe9e2c54c , without lowerdir , whole overlayfs does not make sense?

I don't think this is correct.
Overlay::writable(
                        iter::once(overlay.toolchain_dir.as_path()),
                        upper_dir,
                        work_dir,
                        &target_dir,
                        // This error is unfortunately not `Send+Sync`
                    )

has one lower dir represented by the iterator as required API desc

Possibly relevant https://github.com/containers/podman/blob/master/troubleshooting.md#15-rootless-podman-build-fails-when-using-overlayfs

Demo to showcase the issue for faster test cycles https://github.com/drahnr/overlay-fs-gen

Even the test binary fails with non root (even on the host!), so it has to be a privileged container that is run as root or via the docker daemon.

Since this is about CI, it should be linked with #1
I've been thinking about our options, the most obvious is services: in gitlab CI. It's about providing a network-accessible service via a legacy docker's --link option, in a slow dind mode. It's not possible to run arbitrary code in a "service" interactively, but it's real to overwrite its ENTRYPOINT and CMD, to run something as a server. This is mostly used for serving databases.

Other options to run docker in the pipeline include running a shell executor, binding a socket, but they won't be optimal for us.
The remaining options that I can think of right now:

run the service and the client inside of the same image, the major downside of it is we won't test networking here
have a kubernetes deployment

Other options to run docker in the pipeline include running a shell executor, binding a socket, but they won't be optimal for us.

I think this is how it was done where I worked in the past, now that you put the links up.

* run the service and the client inside of the same image, the major downside of it is we won't test networking here

Not sure how we would test networking right now. That would need some additional machinery.

* have a kubernetes deployment

Yikes, that seems a bit of an overkill.

Idea: Docker can run arbitrary executors, and it might make sense to look into using firecracker or qemu instead of the default runc underlying executor of docker. On the other hand, there is nothing stopping us from using that directly instead of podman in the mentioned PR. It would add a requirement that the host has virtualization/hypervisor extensions enabled.

Other options to run docker in the pipeline include running a shell executor, binding a socket, but they won't be optimal for us.

I think this is how it was done where I worked in the past, now that you put the links up.

* run the service and the client inside of the same image, the major downside of it is we won't test networking here

Not sure how we would test networking right now. That would need some additional machinery.

* have a kubernetes deployment

Yikes, that seems a bit of an overkill.

Idea: Docker can run arbitrary executors, and it might make sense to look into using firecracker or qemu instead of the default runc underlying executor of docker. On the other hand, there is nothing stopping us from using that directly instead of podman in the mentioned PR. It would add a requirement that the host has virtualization/hypervisor extensions enabled.

Not sure how we would test networking right now. That would need some additional machinery.

I guess you mean, e.g. one another process in container acting as proxy, through which those parts would be talking through, to monitor/analyse etc network through that proxy?
(Eventually instrumenting network calls via LD_PRELOAD :P)

Not sure how we would test networking right now. That would need some additional machinery.

I guess you mean, e.g. one another process in container acting as proxy, through which those parts would be talking through, to monitor/analyse etc network through that proxy?
(Eventually instrumenting network calls via LD_PRELOAD :P)

I guess you mean, e.g. one another process in container acting as proxy, through which those parts would be talking through, to monitor/analyse etc network through that proxy?

Yes, but we should discuss that in a separate issue and it's also a topic for the distant future.

I guess you mean, e.g. one another process in container acting as proxy, through which those parts would be talking through, to monitor/analyse etc network through that proxy?

Yes, but we should discuss that in a separate issue and it's also a topic for the distant future.

kubernetes

Yikes, that seems to be a bit of an overkill?
Even though it might look like an overkill (and as a good-enough setup I'd name docker-compose), but kubernetes is more versatile and flexible, so we will win with it in the long term. Besides, making GitLab work with kubernetes is easier than maybe everything else, and certainly more stable.

With Kubernetes we will be able to scale the setup and perform everything we will ever need, like testing networking, distributed multi-source caching etc.

kubernetes

Yikes, that seems to be a bit of an overkill?
Even though it might look like an overkill (and as a good-enough setup I'd name docker-compose), but kubernetes is more versatile and flexible, so we will win with it in the long term. Besides, making GitLab work with kubernetes is easier than maybe everything else, and certainly more stable.

With Kubernetes we will be able to scale the setup and perform everything we will ever need, like testing networking, distributed multi-source caching etc.

I just realized that we are blocked here.
In any case, we will need the green CI, so we'll be able to produce a containerized sccache to run it in CI with dist tests.

I just realized that we are blocked here.
In any case, we will need the green CI, so we'll be able to produce a containerized sccache to run it in CI with dist tests.


Code Release

May I ask when the code can be released? I have a class assignment related to this and I'd like to refer to your excellent work. Thank you

If it is not convenient to make it public, could you send it to me privately?
My email is<EMAIL_ADDRESS>which is used frequently.

Thanks for your interest, code is released.

Dear Liang Chen,
Congratulations to your brilliant achievement and thanks again for your detailed code as well as readme file released in the github.
I'm Yang Jiacheng from Tongji University, and during this shcool year I'm also doing some work in this area due to a class tutored by Phd. Fu Changhong.
Due to my Low-grade equipment, I can hardly make the train code run or even successfully implement it.
I would appreciate it if you could send the trained model file(eg. pth file) to me, and I swear not to use this model anywhere else.
Thanks for your previous reply, and I'm looking forward to keeping in touch with you in the future for further study in academic or other fields.
Best wishes,
Yang Jiacheng
------------------ 原始邮件 ------------------
发件人:                                                                                "liangchen527/SLADD"                                                                                @.>;
发送时间: 2022年5月29日(星期天) 下午4:04
@.>;
<EMAIL_ADDRESS>主题: Re: [liangchen527/SLADD] Code Release (Issue #2)
Closed #2 as completed.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


Create permuted grids when initializing datasets

Reported by @AnsleyManke on 13 Feb 2012 23:05 UTC
Here's an example header showing the first part of the file discussed in Gary Strand's Feb 8,2012 question on the Ferret Users List. Note how lat_vertices and lon_vertices have dimensions (j, i, vertices) with vertices listed to the right of j,i.

ncdump -h thkcello_fx_CCSM4_lgm_r0i0p0.nc
netcdf thkcello_fx_CCSM4_lgm_r0i0p0 {
dimensions:
lev = 60 ;
j = 384 ;
i = 320 ;
bnds = 2 ;
vertices = 4 ;
variables:
double lev(lev) ;
lev:bounds = "lev_bnds" ;
lev:units = "m" ;
lev:axis = "Z" ;
lev:positive = "down" ;
lev:long_name = "ocean depth coordinate" ;
lev:standard_name = "depth" ;
double lev_bnds(lev, bnds) ;
int j(j) ;
j:units = "1" ;
j:long_name = "cell index along second dimension" ;
int i(i) ;
i:units = "1" ;
i:long_name = "cell index along first dimension" ;
float lat(j, i) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude coordinate" ;
lat:units = "degrees_north" ;
lat:bounds = "lat_vertices" ;
float lon(j, i) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude coordinate" ;
lon:units = "degrees_east" ;
lon:bounds = "lon_vertices" ;
float lat_vertices(j, i, vertices) ;
lat_vertices:units = "degrees_north" ;
float lon_vertices(j, i, vertices) ;
lon_vertices:units = "degrees_east" ;
float thkcello(lev, j, i) ;
thkcello:standard_name = "cell_thickness" ;
thkcello:long_name = "Ocean Model Cell Thickness" ;
thkcello:units = "m" ;
thkcello:original_name = "dz" ;
thkcello:comment = "dz at each gridcell" ;
thkcello:original_units = "centimeters" ;
...
}

comment by steven.c.hankin
I'd encourage considering the following approach, which should be much lower impact on the code than rethinking grid definitions to support dimensions before X. Namely, that Ferret supports the idea of "permuted" variables. If this variable had a grid of lon (x) by lat (y) by vertex (z) it would cause no problems at all. I think the current code should see the lon_vertices and lat_vertices variables as permutations of this acceptable grid. I suspect that there is simply a bug in the permutation detection code. Try a grep on "permute" and you should find the relevant code pretty quickly.
Note that one limitation (I believe) in the current code is that when you output the variable to a new netCDF file, it does not remember and restore the original permutation.
(Good to get familiar with this part of the Ferret code, anyway, because it will be where one of the more technically challenging changes needed to support 6D will be found.)
Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1917

This is done, some time back.  As long as the directions of the dimensions are known (from axis attributes for example) the grid of the "vertex" variable is created correctly.


Update dotnet-developer-projects.md

I've listed the project DeveelDB in the category "Data Access"

Thanks! Appreciated.


Android App

Check out the discussion in   https://github.com/Brightside-Social/brightside/issues/2 which has a lot of ideas for the mobile app.
Also the wiki here: https://github.com/Brightside-Social/brightside/wiki/Brightside-Mobile

@stuk88 feel free to break this issue into smaller issues to put on the projects board

Close in favor of new, more granular issues


Data Preprocessing

I wanted to train SPVCNN on my own datasets which were cad files for part segmentation.
First of all, I transformed cad files to point clouds and the format of point clouds were [point clouds numbers, xyz].
Secondly, I normalized the point clouds and I used the sparse_quantize to transform these data.
Data Preprocessing
pc_coords = [pc_numbers, xyz]
point_set = np.concatenate([pc_coords, coords_norm], axis=1)

coords, features = point_sets[:, :3], point_set
coords -= np.min(coords, axis=0, keepdims=True)
coords, indices, inverse_inds = sparse_quantize(coords, voxel_size, return_index=True, return_inverse=True)

coords = torch.tensor(coords, dtype=torch.int)
features = torch.tensor(features[indices], dtype=torch.float)
labels_inds = torch.tensor(labels[indices], dtype=torch.long)

input_data = SparseTensor(coords=coords, feats=features)
target = SparseTensor(coords=coords, feats=labels_inds)
ori_pc_label = SparseTensor(coords=pc_coords, feats=labels)
inverse_map = SparseTensor(coords=pc_coords, feats=inverse_inds)

feed_dict = {'input': input_data, "label": target, "ori_pc_label": ori_pc_label, "inverse_map": inverse_map}

Then, importing sparse_collate_fn from torchsparse.utils.collate and setting collate_fn=sparse_collate_fn in PyTorch DataLoader.
Hence, the input data format of SparseTensor before sending to SPVCNN was [Features (6 dimensions and they are xyz and xyz_normalization), Coordinates (4 dimensions and they are x, y, z and batch index)] after the collate_fn of PyTorch DataLoader.
Results
I have written the code of data preprocessing and training but the result was weird. I used the inverse_map to make outputs map to point clouds and calculate the accuracy with original point clouds' groundtruth.
However, the accuracy of SPVCNN was much lower than PVCNN's result.
Is there any mistake in my data preprocessing?
I hope someone could help me.
Thanks!

Hi @tomanick, thanks for your detailed summary. One thing to keep in mind is that SPVCNN is not always better than PVCNN. This is detailed in the journal version of our paper (https://arxiv.org/abs/2204.11797, see Table 2). The main takeaway is that SPVCNN is suitable for large scenes, while PVCNN is suitable for small objects.

OK. Thanks for your reply.
In addition, I found that if point cloud numbers are not very large like 2048 or 4096, it is not necessary for those data to do sparse_quantize for data-preprocessing.
After doing sparse_quantize, the accuracy of SPVCNN became lower.
When I removed sparse_quantize, the accuracy became better, but it was still lower than PVCNN's result.


[GOBBLIN-402] Add more metrics for gobblin cluster and fix the getJobs slowness issue

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-402] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-402

Description

[x] Here are some details about my PR, including screenshots (if applicable):

Add executor metrics for gobblin cluster job scheduler.
numActiveJobs is optimized to prevent loading job specs every time when sensor is polling.
Add jobQueue metrics in spec consumer

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Local integration test

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@htran1 Please review


Feature: support knife-zero

knife-zero is a promising replacement of knife-solo based on chef-zero instead of custom rsyncing logic+chef-solo.
knife-zero spins up a local chef-zero instance and tunnels its port to the node on demand (e.g. bootstrap or converge), where a regular chef-client runs against this tunneled chef-server. Ohai data gets persisted on the local filesystem and features like search and attribute persistence work out of the box. (even chef-vault may work since knife-zero generates keys for each client (but does not verify them/which is not relevant in chef-vault's use case))
I think it would be a great enhancement to this knife plugin to support bootstrapping with knife-zero besides chef-server and  knife-solo.

@rmoriz I just pushed to master support for knife-zero. I was really surprised how easy it was which scares me. Could you possibly test this out before I cut a release with it to make sure it really works?
This is what I used and it appears to be working.
be knife digital_ocean droplet create --zero -N knifezero.gregf.org -L nyc3 -S 512mb -K $DIGITALOCEAN_SSH_KEY_IDS -I debian-7-0-x64 -r 'recipe[apt]'

Hm, i'm getting TypeErrors:
$ knife digital_ocean droplet create --zero -N test.moriz.net -L ams3 -S 512mb -K 4318 -I debian-7-0-x64 -r 'recipe[apt]'
Droplet creation for test.moriz.net started. Droplet-ID is 4527851
Waiting for IPv4-Addressdone
IPv4 address is: 128.199.52.xxx
Waiting for sshd:done
Doing old-style registration with the validation key at ...
Delete your validation key in order to use your user credentials instead

Connecting to 128.199.52.xxx
ERROR: TypeError: no implicit conversion of nil into String

I can't reproduce... Seems to work here using your same settings.
be knife digital_ocean droplet create --zero -N test.moriz.net -L ams3 -S 512mb -K 42795 -I debian-7-0-x64 -r 'recipe[apt]'
Droplet creation for test.moriz.net started. Droplet-ID is 4548424
Waiting for IPv4-Address.done
IPv4 address is: <IP_ADDRESS>
Waiting for sshd:done
Doing old-style registration with the validation key at /home/gregf/.chef/gregf-validator.pem...
Delete your validation key in order to use your user credentials instead

Connecting to <IP_ADDRESS>
Enter passphrase for /home/gregf/.ssh/id_rsa:
<IP_ADDRESS> Installing Chef Client...
<IP_ADDRESS> --2015-03-21 18:41:21--  https://www.chef.io/chef/install.sh
<IP_ADDRESS> Resolving www.chef.io (www.chef.io)... <IP_ADDRESS>
<IP_ADDRESS> Connecting to www.chef.io (www.chef.io)|<IP_ADDRESS>|:443... connected.
<IP_ADDRESS> HTTP request sent, awaiting response... 200 OK
<IP_ADDRESS> Length: 18358 (18K) [application/x-sh]
<IP_ADDRESS> Saving to: `STDOUT'
<IP_ADDRESS>
100%[======================================>] 18,358      --.-K/s   in 0.002s
<IP_ADDRESS>
<IP_ADDRESS> 2015-03-21 18:41:22 (9.67 MB/s) - written to stdout [18358/18358]
<IP_ADDRESS>
<IP_ADDRESS> Downloading Chef 12 for debian...
<IP_ADDRESS> downloading https://www.chef.io/chef/metadata?v=12&prerelease=false&nightlies=false&p=debian&pv=6&m=x86_64
<IP_ADDRESS>   to file /tmp/install.sh.2397/metadata.txt
<IP_ADDRESS> trying wget...
<IP_ADDRESS> url      https://opscode-omnibus-packages.s3.amazonaws.com/debian/6/x86_64/chef_12.1.1-1_amd64.deb
<IP_ADDRESS> md5      ec2f1b6023c28ac801a46f574eaaacd4
<IP_ADDRESS> sha256   36352c91c213517ee83f9f6e99b992e660e8ea6f256228bd88b62fc253e23a96
<IP_ADDRESS> downloaded metadata file looks valid...
<IP_ADDRESS> downloading https://opscode-omnibus-packages.s3.amazonaws.com/debian/6/x86_64/chef_12.1.1-1_amd64.deb
<IP_ADDRESS>   to file /tmp/install.sh.2397/chef_12.1.1-1_amd64.deb
<IP_ADDRESS> trying wget...
<IP_ADDRESS> Comparing checksum with sha256sum...
<IP_ADDRESS> Installing Chef 12
<IP_ADDRESS> installing with dpkg...
<IP_ADDRESS> Selecting previously unselected package chef.
(Reading database ... 26343 files and directories currently installed.)
<IP_ADDRESS> Unpacking chef (from .../chef_12.1.1-1_amd64.deb) ...
<IP_ADDRESS> Setting up chef (12.1.1-1) ...
<IP_ADDRESS> Thank you for installing Chef!
<IP_ADDRESS> Starting first Chef Client run...
<IP_ADDRESS> Starting Chef Client, version 12.1.1
<IP_ADDRESS> Creating a new client identity for test.moriz.net using the validator key.
<IP_ADDRESS> resolving cookbooks for run list: ["apt"]
<IP_ADDRESS> Synchronizing Cookbooks:
<IP_ADDRESS>   - apt
<IP_ADDRESS> Compiling Cookbooks...
<IP_ADDRESS> Converging 8 resources
<IP_ADDRESS> Recipe: apt::default
<IP_ADDRESS>   * execute[apt-get-update] action run
<IP_ADDRESS>     - execute apt-get update
<IP_ADDRESS>   * execute[apt-get update] action nothing (skipped due to action :nothing)
<IP_ADDRESS>   * execute[apt-get autoremove] action nothing (skipped due to action :nothing)
<IP_ADDRESS>   * execute[apt-get autoclean] action nothing (skipped due to action :nothing)
<IP_ADDRESS>   * apt_package[update-notifier-common] action install
<IP_ADDRESS>     - install version 0.99.3debian11 of package update-notifier-common
<IP_ADDRESS>   * execute[apt-get-update] action run
<IP_ADDRESS>     - execute apt-get update
<IP_ADDRESS>   * execute[apt-get-update-periodic] action run (skipped due to only_if)
<IP_ADDRESS>   * directory[/var/cache/local] action create
<IP_ADDRESS>     - create new directory /var/cache/local
<IP_ADDRESS>     - change mode from '' to '0755'
<IP_ADDRESS>     - change owner from '' to 'root'
<IP_ADDRESS>     - change group from '' to 'root'
<IP_ADDRESS>   * directory[/var/cache/local/preseeding] action create
<IP_ADDRESS>     - create new directory /var/cache/local/preseeding
<IP_ADDRESS>     - change mode from '' to '0755'
<IP_ADDRESS>     - change owner from '' to 'root'
<IP_ADDRESS>     - change group from '' to 'root'
<IP_ADDRESS>
<IP_ADDRESS> Running handlers:
<IP_ADDRESS> Running handlers complete
<IP_ADDRESS> Chef Client finished, 5/5 resources updated in 13.526675851 seconds

Hi, Thank you for trying to support knife-zero!
I'd got same problem.
I found that It fails when trying to expand ssh_command. https://github.com/higanworks/knife-zero/blob/master/lib/chef/knife/zero_bootstrap.rb#L119
I'll try to fix it.

Thanks again for the patch, I'll try to cut a release with knife-zero support later today.

Hi, @rmoriz @gregf
I've released knife-zero 1.3.0 today. https://github.com/higanworks/knife-zero
It doesn't use specific template by bootstraping no longer such as chef-full-localmode.
We can use the all templates that is included with the original Chef.

:+1:
Thank you very much both of you :-)

Hi, Has there been any issue with the release ?
I'll test this feature and write an article my blog to introduce in japan.

@sawanoboly just me slacking. I'll try to get it out this weekend sometime.

@sawanoboly I was hoping to get this out tonight, I updated the documentation, but the tests are severely broken so it may be till next weekend before I can get this done. I won't be around tomorrow to work on it, sorry for the delay.

@sawanoboly sorry for the spam, false alarm about the tests, wasn't as bad as I thought initially. 2.4.0 is released enjoy!


Support Allele Registry CACN IDs in gene and variant curation

CAR has implemented new CACN IDs that represent CNV variants. We need to update the GCI/VCI so that it can accept these new IDs.
Example of new CACN ID in the CAR: https://reg.clinicalgenome.org/allele/CACN1537613926?overlappingGenes=true
https://broadinstitute.atlassian.net/browse/CSP-70
In the VCI:

When adding a new variant record. Currently, the choice is ClinVar ID and or CA ID... this will need to be expanded to include CACN ID as an option.

Note: if a CACN ID is added then a disclaimer should appears that says
"Please note: The VCI is not optimized for the curation of CNVs"

Variant title - needs to be updated so that title can be based on new CACN title (for variant title use "communityStandardTitle" from CAR).

Need to consider knock-on effect to how data is retrieved or displayed in other parts of the VCI. e.g. what to show for functionalities that require a genomic location.

In the GCI:

Everywhere that a variant is currently added, then CACN ID  need to be additional options.

For example, places where variant is saved:

Variant title -  for displays. Same as the VCI, e.g. "communityStandardTitle" from CAR

For example...places where variant title is currently displayed:

Add warning message when someone selects to add a CACN ID:
Note: in general you should be scoring copy number variants involving only one gene, and should not be scoring multi-gene copy number variants due to the fact that the contributions of the other genes cannot be ruled out. If you have any questions please discuss with your expert panel.
When a CACN is entered if it's already in ClinVar then add the ClinVar VariationID instead.

This ticket replaced https://github.com/ClinGen/clincoded/issues/2078

Dependency on Allele Registry to provide:

external IDs, such as ClinVar VariationIDs (to prevent duplicate entries)
standardized title for the variant
list of genes associated with variant

Example: https://reg.clinicalgenome.org/allele/CACN1537613926?overlappingGenes=true
[
{
"@id": "https://reg.genome.network/allele/CACN1537613926",
"CACNid": "CACN1537613926",
"communityStandardTitle": "GRCh38 1p36.31(chr1:5554106-5969826)x4",
"genomicAlleles": [
{
"chromosome": "1",
"coordinates": {
"copies": 4,
"cytoband": "1p36.31",
"end": 5969826,
"start": 5554106
},
"descriptor": "GRCh38 (chr1:5554106-5969826)x4",
"referenceGenome": "GRCh38"
}
],
"overlappingGenes": [
{
"@id": "https://rest.genenames.org/fetch/symbol/MIR4689",
"symbol": "MIR4689"
},
{
"@id": "https://rest.genenames.org/fetch/symbol/NPHP4",
"symbol": "NPHP4"
}
]
}
]

Structure now has name, and overlapping genes... just missing ClinVar ID.

When someone goes to add a variant, in all the locations described above.... then the pop-up modal "Select Variant" should say the following underneath when data is retrieved for a CACN ID:
Below are the data from the ClinGen Allele Registry for the CACN ID you submitted. Select "Save" below if it is the correct variant, otherwise revise your search above:
communityStandardTitle
CACN ID (links to CAR)
ClinVar Variation ID  (blank if not present)
Cytoband
Overlapping Genes
For reference, here is a CA ID output... below.... note... for CACN IDs...

the text is changed in each location from CA ID to CACN ID
the title comes from same source and is still included (e.g. communityStandardTitle)
HGVS Terms are replaced by Cytoband
Additional field of overlapping genes list is added

Note: in general you should be scoring copy number variants involving only one gene, and should not be scoring multi-gene copy number variants due to the fact that the contributions of the other genes cannot be ruled out. If you have any questions please discuss with your expert panel.

Replace "View variant evidence in Variant Curation Interface" link with:
Note: in general you should be scoring copy number variants involving only one gene, and should not be scoring multi-gene copy number variants due to the fact that the contributions of the other genes cannot be ruled out. If you have any questions please discuss with your expert panel.
Add the same text above to the modal, under the "Overlapping Genes" section.
Count the number of genes Overlapping Genes returned, and create a new field called "Number of Overlapping Genes" and situate this above where the "View variant evidence in Variant Curation Interface" link was (above the new text, below the title).
In the Evidence summaries, put the number of overlapping genes in parentheses after the title.
@markmandell

Pasting email sent to Tristan, Phil, and Terry regarding publish messages sent to website (continuing conversation in this ticket):

We are currently working on supporting the Allele Registry CACN IDs in the GCI. I wanted to reach out to you all to confirm how the CACN IDs/Copy Number Variants should be included in the GCI Publish messages. Attached is a full publish message from my dev instance where some CACN IDs are added as Associated Variants to a Gene Disease Record (also added a variant via ClinVar ID to compare the variants).

The additional CNV fields are cacnId and overlappingGenes, and the value for variantType will be “copy number gain”. The overlappingGenes array can be quite large, in this example there are over 100 genes listed for a given variant. My understanding is that the users only want to know how many overlapping genes there are for CNVs and would not need the full list of genes as it is being sent over now. I wanted to propose sending a field with just the count named numberOverlappingGenes or overlappingGenesCount, or appending the count to the variant title to minimize the amount of work needed on your end, but let me know what you think and we can go from there.

Resource ID for dev message: 439fbd7a-0762-4b2e-b8f2-7524c616ee1a
Offset: 154

From @markmandell via email:

Hi all,
We are currently working on supporting the Allele Registry CACN IDs in the GCI. I wanted to reach out to you all to confirm how the CACN IDs/Copy Number Variants should be included in the GCI Publish messages. Attached is a full publish message from my dev instance where some CACN IDs are added as Associated Variants to a Gene Disease Record (also added a variant via ClinVar ID to compare the variants).
The additional CNV fields are cacnId and overlappingGenes, and the value for variantType will be “copy number gain”. The overlappingGenes array can be quite large, in this example there are over 100 genes listed for a given variant. My understanding is that the users only want to know how many overlapping genes there are for CNVs and would not need the full list of genes as it is being sent over now. I wanted to propose sending a field with just the count named numberOverlappingGenes or overlappingGenesCount, or appending the count to the variant title to minimize the amount of work needed on your end, but let me know what you think and we can go from there.
Happy to discuss via Slack or we can continue through this email thread. Thanks and let me know if you have any questions or if additional information is needed.
Link to the ticket: https://github.com/ClinGen/gene-and-variant-curation-tools/issues/32
Best,
Mark Mandell

Few questions:

I see 'copy number gain' as a variation type. Would 'copy number loss' also be valid? There is a parallel discussion on this going on in GA4GH VRS about how to describe copy number variation: (https://github.com/ga4gh/vrs/issues/404#issuecomment-1419363538). As long as the valid options are in the range of the EFO or SO terms listed there, we're good.
Overlapping genes are represented by their symbols; good for readability, less good for computability (though I see that's the form they're coming over in from the Allele Registry). I'm assuming it would be a bother to use HGNC, Ensemble, or NCBI Entrez IDs?
There's no structured description of the CNV, I think (affected coordinates, variable breakpoints, copy number). I am assuming we would have to go to either the Allele Registry or ClinVar for this info (or parse the provided expression)?

Hi @tnavatar,

Going to review the discussion you linked and check for variationType of "copy number loss" as you mentioned. Will also check in with Matt to be sure.
For the overlapping genes list, there is a reference link to XML for each gene provided by the Allele registry that I have omitted from being saved to our DB when a user adds the CNV. Right now, we are only saving the list of symbols for display of symbols and gene count in the UI. However, I can see if parsing out the hgnc_id from the XML for each gene would be feasible, if that is something you are interested in. Although could be a fairly big list for some variants (some over 100, but not sure if thats a concern on your end). So could be:

overlappingGenes: [
  {
    "geneSymbol": "ACTG1P12",
    "hgnc_id: "HGNC:44496"
  },
  ....
]

The data in genomicAlleles can also be parsed and sent over as well, similar to what we do for CA IDs for hgvs names. Let me know if what I included below is what you are mentioning. Can either parse out the relevant info or include as is.

    "genomicAlleles": [
      {
        "chromosome": "3",
        "coordinates": {
          "copies": 3,
          "cytoband": "3p26.3-3p24.3",
          "end": 19510600,
          "start": 63843
        },
        "descriptor": "GRCh38 (chr3:63843-19510600)x3",
        "referenceGenome": "GRCh38"
      }
    ],

Here is a link to the example I am using in this comment for reference https://reg.clinicalgenome.org/allele/CACN1538018649?overlappingGenes=true

@tnavatar
For us CACN IDs are like CA IDs, in that we primarily need them to provide a unique identifier for the variant object within the VCI/GCI... .we also need a title for displaying the variant...and the genomic coordinates/HGVS are also used for bringing in associated evidence in the VCI. When a curators goes to add in a variant, we also bring in some associated data on-the-fly that allows the user to make sure they have added the correct variant (e.g. alternate HGVS titles in the VCI). For CACN IDs in the GCI, this added curator help includes how many genes overlap the region. This is important because curators should only use a CACN ID if there is only ONE gene in the repeat region. If there is more than one then they should not be adding it. So, I think it is unlikely you will get a long string of genes published to the website with a CACN ID.
HGNC IDs
We get the list of overlapping genes from CACN ID JSON output from the CAR API on-the-fly,  and this does not currently include the HGNC IDs (just the gene symbols)...if you need them then there are probably 2 routes for this, 1) we ask Baylor to include them in the CACN JSON output, then we add re-code our GCI outputs, and you work with us to receive these new fields, or 2) you use the CACN IDs to get this data directly from the CAR.  Or perhaps, on the website you only need to list the number of overlapping genes? For reference, in the evidence summary in the GCI users said they only wanted to see the number of overlapping genes, not the whole list of what the genes are... maybe the request is the same for the website? And we have the number of overlapping genes and so could just provide that to you... and it could be added as a separate field, or just appended in parentheses to the end of the variant title? So, you may not need to list the genes at all?
CACN type
With respect to whether the the CACN is a "copy number loss" or "copy number gain". I think the only way to know this is if someone looks at the copy number and compares it to the reference genome? It is listed in ClinVar entries but not in the CAR generally for CACN IDs, so this is not information we would have for every CACN ID.
@ErinRiggs and @courtneythaxton  - it would be good to know which of the fields being suggested is required or not (e.g. the HGNC ID for the overlapping gene, and whether it is an loss/gain type CACN)....i.e.  it would be good to get clarification on whether these would be useful/necessary for the users to see on the website and whether you feel we should hold off on the release of this feature until we have these fields.

Hey @tnavatar, given the above conversation, I was wondering if the current output we are sending is sufficient for now until we discuss adding scope for future use. So would be 2 additional fields being added for CNVs in GCI: overlappingGenes and cacnId. Let me know what you think and we can coordinate from there. Thanks!

Hi All,
Thanks for starting this discussion thread Matt.
I agree with what has been mentioned thus far and just wanted to add a few comments.

It is true that a CACN with multiple genes reflects a multigene disorder which does not fit the current monogenic framework, however it may be useful to publish the case with a score of 0, especially if the paper suggested it was a single gene, only for the mapping to indicate it was not.
*   In these cases I think having the # of overlapping genes would suffice for the report, but I would especially like if the genes are listed in the CAR, which I believe they will be. I think the website and GCI/VCI can then use linking to the CAR page for the region as a way a user can see those genes.
The copy number might be good to see, I agree, but I think we may be able to put this aside for a future iteration to the website and/or GCI/VCI.

Thanks for all the hard work on this!
Courtney
--
Courtney Thaxton, Ph.D. (she/her/hers)
Director, ClinGen-affiliated UNC Biocuration and Coordination Core
Research Assistant Professor, Dept. of Genetics
University of North Carolina at Chapel Hill
120 Mason Farm road
5100B Genetic Medicine Building
CB#7264
Chapel Hill, NC 27599
From: tnavatar @.>
Date: Wednesday, March 15, 2023 at 4:25 PM
To: ClinGen/gene-and-variant-curation-tools @.>
Cc: Thaxton, Courtney Lynn @.>, Mention @.>
Subject: Re: [ClinGen/gene-and-variant-curation-tools] Support Allele Registry CACN IDs in gene and variant curation (#32)
@wrightmwhttps://github.com/wrightmw Gotcha--so the list of genes should be less relevant for the purpose of receiving data from the GCI; there should only be one and it should match the gene being curated.
Regarding copy number type, I figure you can infer that for autosomal variants, but when talking about variants on sex chromosomes it's nice to have an explicit descriptor. I don't know that it would matter for showing on the website in its current form, but it would be good to think about getting that for future uses. Seems like something that should be added to the CAR.
—
Reply to this email directly, view it on GitHubhttps://github.com/ClinGen/gene-and-variant-curation-tools/issues/32#issuecomment-1470791098, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AHYF4VMC2CJHOK6IGXLVKJTW4IQSLANCNFSM5DARRMBQ.
You are receiving this because you were mentioned.Message ID: @.***>

It seems like the current output should be OK for display purposes; it sounds like desired changes in the future might be more related to the CAR anyway.

On Mar 17, 2023, at 12:33 PM, Mark Mandell @.***> wrote:
Hey @tnavatar https://github.com/tnavatar, given the above conversation, I was wondering if the current output we are sending is sufficient for now until we discuss adding scope for future use. So would be 2 additional fields being added for CNVs in GCI: overlappingGenes and cacnId. Let me know what you think and we can coordinate from there. Thanks!
—
Reply to this email directly, view it on GitHub https://github.com/ClinGen/gene-and-variant-curation-tools/issues/32#issuecomment-1474100388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5MMGDAKDRHNENGMF6QVLW4SG47ANCNFSM5DARRMBQ.
You are receiving this because you were mentioned.

Hi @tnavatar, wanted to let you know that we released the support of CACN IDs in the GCI today. Let me know if you have any questions or need anything from us. Thanks!!


map the patient for visualisatation of chain

graphically map the patients ,area wise dipicting possible chain through which it spread .which will allow peple to know the affected people in thier area and and possibly notice the chain it could have spread. therefore maintaining the distance from suspected people.
also it would help people who are showing some symptoms to go in self qurantine if they are facing some symptoms and assume them to be somewhere in the chain

Data provided by the state govts is not enough to carry this development out.


feat. helmfile render

I am enjoying some of the advanced features, including referencing other helmfiles with the configuration:
helmfiles:
  - path: to/other/manifest/syntax

akin to helm template chart/ might do, I would find it helpful for debugging to have a similar subcommand for helmfile to print the entire manifest with releases across files combined. Additionally, any templating on the manifest could be done as well at this stage.

or maybe call the subcommand expand to take vocab from source

yeah, definitely. debugging this is a nightmare

print the entire manifest with releases across files combined

What do you mean by this? If you literally meant "manifest", you should be able to use helmfile template to render all the K8s resources across all the sub-helmfiles.
Otherwise try helmfile build.

build is perfect, thanks!

sorry @mumoshu you're right helmfile build does what I am looking for - I got hung up on helmfile template doing more than I needed when pulling in sub-helmfiles.


update spectral_factor_firstsolar

Main change: update data screens in spectrum.spectral_factor_firstsolar

[X] Closes #2086 , relates to project #2065
[X] I am familiar with the contributing guidelines
[X] Tests added
[ ] Updates entries in docs/sphinx/source/reference for API changes. Not necessary?
[ ] Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
[X] New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
[ ] Pull request is nearly complete and ready for detailed review.
[ ] Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

Summary of changes so far following discussion in issue #2086:

Added parameters to enable user to set min/max airmass
Changed handling of values exceeding max pw to default to max pw rather than np.nan
Replaced subjective phrasing such as "exceptionally high" with "high" and removed references to "divergence" in the warnings in order to align with the fact that the user can now determine their own limits, which may or may not be "exceptionally" high/low or cause model divergence
Added tests for new AMa limit parameters, and modified warnings and expected values in existing tests

These are the proposed changes; more than happy to modify/add/retract following reviews. If anyone has any views differing from the conclusion reached through the discussion in #2086 that could impact the proposed changes then please share.

@kandersolar is the failed test the one that you mentioned before fails randomly sometimes?

@kandersolar is the failed test the one that you mentioned before fails randomly sometimes?

@RDaxini Yes it's unrelated to your contribution.

removed references to "divergence" in the warnings

I would rather see this information moved to a Notes section, than deleted. The reason for the bounds should be retained. For PW, the bounds ensure that the model doesn't "diverge" from physically reasonable values, e.g., if pw=-0.01 then the last term involving $\frac{1}{\sqrt{pw}}$ could cause the modifier to fall outside of [0, 1].

 Based on the PVLIB Matlab function ``pvl_FSspeccorr`` (...)

Is this statement referring to the matlab function really necessary?

 Based on the PVLIB Matlab function ``pvl_FSspeccorr`` (...)

Is this statement referring to the matlab function really necessary?

Not any longer, this reference can be removed.

All good from my side now save for this one comment --- is the whatsnew entry for this PR okay?

Thank you @RDaxini for some great GSoC contributions!


Getting exception 'This socket is closed'

Error: This socket is closed
at WriteStream.Socket._writeGeneric (net.js:679:19),
at WriteStream.Socket._write (net.js:730:8),
at doWrite (_stream_writable.js:331:12),
at writeOrBuffer (_stream_writable.js:317:5),
at WriteStream.Writable.write (_stream_writable.js:243:11),
at WriteStream.Socket.write (net.js:657:40),
at module.exports (/<MYPROJECT>/node_modules/sticky-cluster/lib/log.js:11:18),
at Object.entrust (/<MYPROJECT>/node_modules/sticky-cluster/lib/workers.js:58:16),
at Server.<anonymous> (/<MYPROJECT>/node_modules/sticky-cluster/lib/master.js:14:17),
at emitOne (events.js:96:13),
at Server.emit (events.js:188:7),
at TCP.onconnection (net.js:1464:8)

This is happening when I am keeping server as idle I am seeing this exception after 30 minutes.
I am using winston for logging and overriden console.log also
console.log = function() {
    _logger.info.apply(_logger, Array.prototype.slice.call(arguments));
};

It works for me after more than 40 minutes being idle. Also it's a very strange stack trace.. Try to:
a) set the DEBUG option to false,
b) not overload console.log,
c) double-check any configured timeouts (websocket config, nginx config, etc).

The exception is happening for process.stdout.write,
stdout is not available ? !!

Omg, may be process.stdout is actually an instance of Socket?) It would explain the strange stack trace. Haven't you accidentally redefined it too?

My code also had process forking based on the number of cores.  I think
your library is already doing the same.
Sujith Krishnan
Tech Lead
m<PHONE_NUMBER>
w www.julysystems.com http://julysystems.com/
https://www.facebook.com/Julysystems [image: tw]
https://twitter.com/julysystems [image: li]
https://www.linkedin.com/company/july-systems [image: g]
https://www.youtube.com/channel/UC0c7qtTPSleoBENH_k6DllA
On Thu, Sep 7, 2017 at 1:58 PM, Denis Zhbankov<EMAIL_ADDRESS>wrote:

Omg, may be process.stdout is actually an instance of Socket?) It would
explain the strange stack trace. Haven't you accidentally redefined it too?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/uqee/sticky-cluster/issues/24#issuecomment-327729070,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AS79zSPLVxoFK0-ameEAgE8eGo2gtBEKks5sf6k8gaJpZM4PMjI_
.

--

DISCLAIMER
This email message along with attachments, email thread if any, is for the
sole use of the intended recipients and for the intended purpose and may
contain July Systems confidential and privileged information.Any
unauthorized review, use, disclosure, copying, distribution or any other
form of unintended use is strictly prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all copies
/ forms of the original document.

Yes, it does. You can configure the exact number of workers, which equals the number of cores by default.


"Runtime error: invalid memory address or nil pointer dereference" while navigating

Describe the bug
Golang error that stopped the execution.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x9a8 pc=0x1059375]

goroutine 1 [running]:
github.com/jesseduffield/gocui.(*View).draw(0xc0003a6500, 0xffffffffffffffff, 0xc00018d748)
	/home/circleci/project/vendor/github.com/jesseduffield/gocui/view.go:360 +0x474
github.com/jesseduffield/gocui.(*Gui).draw(0xc0002ea370, 0xc0003a6500, 0x0, 0x0)
	/home/circleci/project/vendor/github.com/jesseduffield/gocui/gui.go:750 +0xe5
github.com/jesseduffield/gocui.(*Gui).flush(0xc0002ea370, 0x0, 0x0)
	/home/circleci/project/vendor/github.com/jesseduffield/gocui/gui.go:535 +0x1ef
github.com/jesseduffield/gocui.(*Gui).MainLoop(0xc0002ea370, 0xc0002ea370, 0x0)
	/home/circleci/project/vendor/github.com/jesseduffield/gocui/gui.go:456 +0x247
github.com/jesseduffield/lazydocker/pkg/gui.(*Gui).Run(0xc00035a580, 0x0, 0x0)
	/home/circleci/project/pkg/gui/gui.go:310 +0x2db
github.com/jesseduffield/lazydocker/pkg/gui.(*Gui).RunWithSubprocesses(0xc00035a580, 0xc0002ee9b0, 0x0)
	/home/circleci/project/pkg/gui/subprocess.go:20 +0x2f
github.com/jesseduffield/lazydocker/pkg/app.(*App).Run(...)
	/home/circleci/project/pkg/app/app.go:54
main.main()
	/home/circleci/project/main.go:74 +0x883

To Reproduce
Cannot reproduce, but i was just navigating from Menus to Menus (logs, stats...), and from services to services.
Expected behaviour
Normal behaviour.
Screenshots

Desktop (please complete the following information):

OS: MacOS X Mojave
Lazydocker Version v0.7.4

looks like when we reference v.viewLines in line 360 of view.go the view no longer exists, probably because it was a menu that closed or something like that. Though we don't actually delete views, we just remove them from the global view array in DeleteView(). So I'm not sure what's going on here. Does anybody have ideas?

This should now be resolved because we no longer delete any views (or remove them from the global view array) as of https://github.com/jesseduffield/lazydocker/pull/372


Question: smoltcp-based client <--> server communication (with 2 tap interfaces)

Networking noob here - Wasn't sure if this was the right place for this question. So, please let me know if this question needs to be directed some place else.
I'm trying to get a smoltcp-based client to talk to a smoltcp-based server but cant seem to get them to talk to each other.
Tried a couple of things so far,

Using 2 Tap Interfaces:

My setup has 2 tap interfaces (tap0 and tap1). One for the client and the other for the server. The client sends raw ipv6 packets to the server and vice-versa. Both interfaces use the same set of instructions (as laid out in the usage examples)
The client is assigned fdaa::1 and the server is assigned fdaa::2.
However, my client can't seem to find the server.

Using a Veth pair:

My understanding is that the examples don't support veth ifaces, as it results in an OS error - Invalid argument, when calling TapInterface ::new()

Any help here would be appreciated! :)
Client logs:
(base) root@DESKTOP-M9O6B7J:/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2# cargo run --example hip_initiator
    Finished dev [unoptimized + debuginfo] target(s) in 6.33s
     Running `target/debug/examples/hip_initiator`
[0.0s] (socket::set): [0]: adding
[0.0s] (rustdhipv2::daemon::hipd): HIP_STATE is: Some(Unassociated)
[0.0s] (rustdhipv2::daemon::hipd): Starting HIP BEX
[0.0s] (rustdhipv2::daemon::hipd): Sending I1 packet
[0.0s] (socket::raw): #0:IPv6:0x8b: buffer to send 96 octets
[0.0s] (rustdhipv2::storage::HIPState): (key, value) pair inserted
[0.0s] (iface::ethernet): non-unicast source address
[0.0s] (iface::ethernet): cannot process ingress packet: malformed packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=4e-c3-e8-b3-84-36 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=:: dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
.....
[0.0s] (hip_initiator): poll error: malformed packet
[0.0s] (iface::ethernet): non-unicast source address
[0.0s] (iface::ethernet): cannot process ingress packet: malformed packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=4e-c3-e8-b3-84-36 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=:: dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: malformed packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=4e-c3-e8-b3-84-36 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=4e-c3-e8-b3-84-36 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdaa::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fdaa::2 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdaa::2 missing, silencing until t+3.000s

My setup has 2 tap interfaces (tap0 and tap1). One for the client and the other for the server. The client sends raw ipv6 packets to the server and vice-versa. Both interfaces use the same set of instructions (as laid out in the usage examples)

Try setting up your host machine to forward packets between those interfaces. I don't quite recall how exactly to do that; I think the easiest way might be to add them to a bridge, but you could try setting up IP forwarding too.

So far, I've tried linking both tap(s) to a bridge (named br0) with brctl (a tool I came across after some googling) .
brctl addbr br0
ip link set br0 up
brctl addif br0 tap0
brctl addif br0 tap1
But haven't added the host interface (i.e. eth0) to the bridge.
To clarify, the suggestion is to add the host interface (i.e. on my machine that's eth0) to the bridge, which should allow packets from tap0 to be routed to tap1 i.e. ( tap0 <--> eth0 <--> tap1) .

This looks like #419 . Can you try with latest master?
(Adding the taps to a bridge should be enough unless you want smoltcp to talk with other machines in your LAN or the outside Internet)

Tried master branch. I get the same result but noticed that the server (or responder) gets an IP --> MAC mapping for the client.
(base) root@DESKTOP-M9O6B7J:/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2# cargo run --example hip_responder
    Finished dev [unoptimized + debuginfo] target(s) in 5.98s
     Running `target/debug/examples/hip_responder`
[0.0s] (socket::set): [0]: adding
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::215:5dff:fe2a:308d dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_responder): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=f2-2b-7a-98-ad-6d dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_responder): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=f2-2b-7a-98-ad-6d dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_responder): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::215:5dff:fe2a:308d dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_responder): poll error: unrecognized packet
[0.0s] (iface::neighbor): filled fdaa::1 => 02-00-00-00-00-02 (was empty)

whereas the client (or initiator) doesn't seem to acquire a similar mapping for the server and I'm not sure why.  Client and server impl(s) aren't all that different.
I'm assuming the client's neighbor solicitation message is received but the server's response for some reason does not make it back.
(base) root@DESKTOP-M9O6B7J:/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2# cargo run --example hip_initiator
   Compiling rustdhipv2 v0.1.0 (/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2)
    Finished dev [unoptimized + debuginfo] target(s) in 1m 49s
     Running `target/debug/examples/hip_initiator`
[0.0s] (socket::set): [0]: adding
[0.0s] (rustdhipv2::daemon::hipd): HIP_STATE is: Some(Unassociated)
[0.0s] (rustdhipv2::daemon::hipd): Starting HIP BEX
[0.0s] (rustdhipv2::daemon::hipd): Sending I1 packet
[0.0s] (socket::raw): #0:IPv6:0x8b: buffer to send 96 octets
[0.0s] (rustdhipv2::storage::HIPState): (key, value) pair inserted
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::215:5dff:fe2a:308d dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fe80::100 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdbb::2 missing, silencing until t+3.000s
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::100 dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (iface::ethernet): cannot process ingress packet: unrecognized packet
[0.0s] (iface::ethernet): packet dump follows:
EthernetII src=12-95-6a-37-a4-41 dst=33-33-00-00-00-16 type=IPv6
\ IPv6 src=fe80::215:5dff:fe2a:308d dst=ff02::16 nxt_hdr=Hop-by-Hop hop_limit=1
[0.0s] (hip_initiator): poll error: unrecognized packet
[0.0s] (socket::meta): #0: neighbor fdbb::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fe80::100 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdbb::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdbb::2 silence timer expired, rediscovering
[0.0s] (socket::raw): #0:IPv6:0x8b: sending 96 octets
[0.0s] (iface::ethernet): address fe80::100 not in neighbor cache, sending Neighbor Solicitation
[0.0s] (socket::meta): #0: neighbor fdbb::2 missing, silencing until t+3.000s
[0.0s] (socket::meta): #0: neighbor fdbb::2 silence timer expired, rediscovering

In case, this is needed, a snapshot of my current setup (hardware: WSL2 machine running ubuntu)
(base) root@DESKTOP-M9O6B7J:/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
....
....
5: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:15:5d:2a:30:8d brd ff:ff:ff:ff:ff:ff
20: tap1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br0 state DOWN mode DEFAULT group default qlen 1000
    link/ether f2:2b:7a:98:ad:6d brd ff:ff:ff:ff:ff:ff
21: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel master br0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 12:95:6a:37:a4:41 brd ff:ff:ff:ff:ff:ff
22: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 12:95:6a:37:a4:41 brd ff:ff:ff:ff:ff:ff

(base) root@DESKTOP-M9O6B7J:/mnt/c/Users/Nil/devspace/rust/projects/rusty_hipv2# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.12956a37a441       no              tap0
                                                        tap1

Update: - in case, anyone happens to come across a similar issue, the following may prove useful.
So, WSL 2 (or windows subsystem for linux) distributions have limited IPv6 networking support. Microsoft is working on the issue. More specifically, you cannot reach IPv6-only addresses from a WSL 2 instance. Although, they haven't explicitly stated whether this applies to tap interfaces, I'm presuming its does.
Anyway, I switched to using IPv4 addresses (after some refactoring as my initial target was IPv6-only). It works as expected now.
Steps to build a linux bridge and wire-up 2 tap interfaces:

brctl addbr br0
ip link set br0 up
brctl addif br0 tap0 tap1

Not sure if WSL is the issue, the traffic here shouldn't leave the Linux kernel.
From a quick look at your code, one possible issue I see is that the two IPs you chose are in different subnets:
    let initiator_addr = IpAddress::from_str("fdaa:0:0:0:0:0:0:1").expect("invalid address format");
    let responder_addr = IpAddress::from_str("fdbb:0:0:0:0:0:0:2").expect("invalid address format");

Try picking 2 addrs in the same subnet. fdaa::1/64 and fdaa::2/64, for example

Using ip addresses within the same subnet was actually one of my first attempts.
Also tried a number of other permutations. But at some point, I noticed a regular ping6 from tap0 to tap1 didn't work either, whereas ipv4 pings worked fine..

I can recommend to use macvtap devices¹ in bridge mode, they are easy to set up and you can also use them to speak to the Internet (only the kernel network stack needs an additional macvtap in bridge mode to be able to talk to smoltcp).
You can also use the veth pair if you setup smoltcp to use a RAW socket on each (and the same IP address) which, however, also requires an iptables rule to prevent the kernel sending RST TCP packets for ports it does not know.
¹ https://github.com/ANLAB-KAIST/usnet_devices (but I need to review the PR first)

I'll try the macvtap option and report back (sometime later, as I'm well into my current project which has switched to using an ipv4 setup).

Using a Veth pair:
My understanding is that the examples don't support veth ifaces, as it results in an OS error - Invalid argument, when calling TapInterface ::new()

My original idea was to use a veth pair as I'm using RAW sockets (with and without same IP addresses) but kept running into the above error.
Note:- I couldn't get iptables working at all in WSL2.

My understanding is that the examples don't support veth ifaces, as it results in an OS error - Invalid argument, when calling TapInterface ::new()

Instead of trying to use the veth interface like a tap interface you would use a RAW socket on top of the veth interface (RawSocket::new("vethX")).

will try this.


Fix package script bug

Previously if you use a custom repo you will see an error about saving the image failed.
This PR corrects that error so that local builds with custom REPO locations will work.

/backport SKIP release/v4.0

/backport SKIP release/v3.0


Detect danger zone

With the ultrassound sensor, we need to detect objects that are within a certain distance that will be declared as something like a danger zone.

This is already implemented in commit b78387a. Now it's only missing the refinement of the code to create the optimal detection zone.

As per this f47d411 this issue is completed and can be closed.


upgrade clap, or move away from it

clap 4.0 is out, and hyperlink is out of date
however, clap 5.0 seems to be around the corner as well
should we just move to the argh crate?  I had never heard of argh before seeing this ticket, so I decided to have a look.  I made branch for each
https://github.com/oylenshpeegul/hyperlink/tree/clap-v4
https://github.com/oylenshpeegul/hyperlink/tree/argh
Changing from clap 3 to 4 wasn't too bad. Let's hope changing to clap 5 will be even easier.

neat. what's the release binary size?

❯ ls -lh target/release/hyperlink
-rwxr-xr-x 2 tim tim 1.9M Oct 28 15:21 target/release/hyperlink

Not much of a difference on my machine
argh
❯ ls -lh target/release/hyperlink
-rwxr-xr-x 2 tim tim 1.9M Oct 28 15:21 target/release/hyperlink
clap-v4
❯ ls -lh target/release/hyperlink
-rwxr-xr-x 2 tim tim 2.2M Oct 28 15:27 target/release/hyperlink
clap-v3
❯ ls -lh target/release/hyperlink
-rwxr-xr-x 2 tim tim 2.2M Oct 28 15:29 target/release/hyperlink

a win is a win, feel free to make a PR for argh, for as long as the output is roughly the same and the CLI doesn't have breaking changes


Bug in the order of some lists

there are some bugs in some of the lists, one of them is on genres -> free to play or genres -> action, it also happens in some tags games like for example tags -> shooter.
I think it happens when there is a lot of game listed because for example if you click in Local Co - Op it works properly.
I would like you to fix it, thank you very much.

Thanks for the report, I've fixed the issue.


Provide multiple modes for navigating imported images.

Imported images might be shown in different styles:

One by one;
List of images with preview;
Table of images.

@FlaviusHouk, propose move this to the v0.2

We can create one view for the version 0.1 and later, expand it.

So in this case we need to change title, because the future functionality will not correspond to it ... I guess. What you think about it?

I think it is fine to leave it as it is. This feature will be implemented now particlularly and fully in future. So title displays what we are going to do.

Ok, no problem.

First mode implemented with commit 9effaf5. Other ones will be implemented in v0.2 after providing setting window.


where is imgui?

I cannot find source file:
../../../../../../third_party/imgui/imgui_demo.cpp

See the game controller README for instructions on where and how to clone the ImGui third party dependency.


Add observability example

fixes #297

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/camel-quarkus-pr/297/

@jamesnetherton ah I also added an example, do you mind resolving the examples/pom.xml conflict

Rebased & resolved conflicts.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/camel-quarkus-pr/301/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/camel-quarkus-pr/304/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/camel-quarkus-pr/305/


Test the "Process Management with CFEngine" article

Test the article and ensure the process works as documented. File any issues a writing issues.
https://docs.docker.com/engine/articles/cfengine_process_management/

#dibs

Documentation seems outdated. Issue opened here:
https://github.com/docker/docker/issues/11703

Can this be written as an e2e or integration test? Haven't seen any of those in the codebase, probably this is meant to be manual, right?
Thx,

@fgimenez Eventually, we'd like any sample code in the documentation to be tested in an automated fashion.  That is a couple years out at the moment. In the mean time, we have to rely on manual testing.
In any case, this issue is about testing the procedure as written which includes more than the actual code.

I can further test this issue and would anyone like me to submit any automated test scripts that I would write for this?

@zelahi also look at https://github.com/docker/docker/issues/11703, which is probably related.
Having scripts to automatically test the example would be cool, but not required to solve this issue (I think @SvenDowideit is also thinking about automating those).
So, just testing the examples, and (if needed) update the documentation is good enough, but if you have suggestions to automatically test the examples, that's always welcome :smile:

sure, It might be better to have the tests as a separate project from the original docker project.   There could be an automated docker testing project, which automates the examples in the docs, then we can have those tests hook up to Jenkins and automate running those tests.  What do you guys think?

@zelahi That pretty much describes the long term project we have.  If you want to contribute such a setup...that would be a nice contribution indeed.

Sure, I don't have a Jenkins server I can contribute but I can figure out some more details and open up and issue with details of this project

Sure, I don't have a Jenkins server I can contribute

@zelahi let me know if you need help with this, I have some spare infra and experience with Github pull request builder plugin
Cheers,

noice :)


The training time of pretrain models

Pretrain upstream models, including Mockingjay, Audio ALBERT and TERA.
How many GPUs and what types of GPUs did these three models use, and how long did they train?

I tried to start the TERA training, adjusted the bs, and used 8*P40(24G) graphics cards. After three days, the training progress was only 3%.
Here is my adjusted config file:
runner:
  n_epochs: -1
  total_steps: 1000000
  gradient_clipping: 5.0
  gradient_accumulate_steps: 8

  log_step: 50000
  save_step: 50000
  max_keep: 5

  fp16: false

optimizer:
  name: AdamW_with_schedule
  lr: 2.e-4
  warmup_proportion: 0.07

pretrain_expert:
  datarc:
    num_workers: 8
    train_batch_size: 128
    max_timestep: -200 # Max length for audio feature (0 for no restriction, negative value to set minimum timestep)
    libri_root: 'datasets/LibriSpeech/' # If raw libri data is provided, use on-the-fly feature extraction, else use the pre-extracted features under `file_path`
    file_path: 'data/len_for_bucket' # Pre-extracted features path. When using on-the-fly feature extraction, this is used to provide length for bucketing.
    sets: ['train-clean-100'] # can be the subset of ['train-clean-100', 'train-clean-360', 'train-other-500']

Hi,
In the TERA paper, we train the model with an actual batch size of 32 (4 * 8) by setting:
gradient_accumulate_steps: 4
train_batch_size: 8

Please also see here:
https://github.com/s3prl/s3prl/blob/540a4f86e9f099b240c314a56ff80b069e91d4cd/s3prl/pretrain/runner.py#L109-L112
This will take approximately 5 days on a single 1080Ti GPU.
Note that gradient_accumulate_steps are virtual training steps for stacking the batch size when GPU memory is limited. After gradient_accumulate_steps amount of virtual steps, the actual training step is increased by one, where the maximum actual training step is defined by total_steps.
Looking at your config, your batch size is set to 128*8=1024, which is very large. The training progress at 3% after three days looks reasonable to me at such a large batch size.
To shorten training time, you can either:

reduce the batch size, by decreasing gradient_accumulate_steps and/or train_batch_size
reduce total_steps: 1000000, since you shouldn't need that many steps for the model to converge if you are training with a large batch size.
use more GPUs so you can reduce the gradient_accumulate_steps.

Andy

Thank you for your answer, I would like to reproduce your results with the same configuration and as fast as possible.
For gradient_accumulate_steps: I would drop it to 1;
For train_batch_size: I increased the bs to improve GPUS utilization and speed up training. In theory, the larger the bs, the faster the speed. In the case of enough video memory, shouldn't the bs be bigger the better?
For total_steps: I thought that N GPUs added up to total_steps, so I think the larger the bs, the more GPUs, which will speed up the training. Am I understanding wrong here?

(Please ignore the following, it's my machine's problem)
The slowness may also be related to my machine.
Due to my machine, it is very slow to fetch data one by one from the scattered file pile through csv. I seem to need to preprocess all the data into one big file and then index it for training, which will be faster.

the paper setting is 8*1000000,

I believe it is a batch size of 32 with 1M total steps.
As long as train_batch_size times gradient_accumulate_steps yields 32 then you are using the same setting as ours, so the following are all correct:
case 1: train_batch_size=32, gradient_accumulate_steps=1
case 2: train_batch_size=16, gradient_accumulate_steps=2
case 3: train_batch_size=8, gradient_accumulate_steps=4
and so on.
In all cases, you should set total_steps=1000000.

If I set bs=8 -> bs = 256, then steps should be 8000000/ 256 = 31250. Rigth?

I understand you want to increase the batch size and thus reduce the training steps,
although the amount of seen samples is the same, please note that the gradients will be different due to the large batch size. (In other words, training the same model with small batch size v.s. large batch size might get very different results.)
That being said, I haven't tried training with a larger batch size so I don't have much to share.
The amount of seen samples of our setting is 32*1000000 and not 8*1000000.
So if you were to increase batch size from 32 to 256, you can reduce total_steps to 1000000/(256/32) = 125000

Thanks,  clearly!
Suppose the GPU has 12G of video memory, but only 2G is used, which is too uncomfortable.
In my past computer vision tasks, the size of the bs is generally adjusted so that the memory usage is close to 11.5/12.
Usually, a large BS will achieve better results than a small BS, and the training speed is also faster. I don't know if this is the case in voice tasks as well.
I'll update again if it's faster after setup.

Thank you for your careful answer, which made it clearer.

And FYI, this model can be trained within 5 days on a single GPU if you use our settings.

I'll give it a try.
By the way, the TERA default config will take approximately 5 days on a single 1080Ti GPU.
How about mockingjay  and AALBert?

Thank you for your careful answer, which made it clearer.
At this rate, I seem to need 25 days instead of 5.

Ah! I think you are right, I forgot an important factor that effects training time, which is the amount of data.
5 days of training time is for training on sets: ['train-clean-100'],
25 days of training time is for training on sets: ['train-clean-100', 'train-clean-360', 'train-other-500'],
see below for more info, this is related to the total_steps it takes for the model to converge for a given amount of data.

By the way, the TERA default config will take approximately 5 days on a single 1080Ti GPU. How about mockingjay  and AALBert?

They are almost the same, since they are all 3-layer transformer encoders, and training objectives are similar.

In addition, now I only used the train-clean-100, the train-clean-360 and train-Other-500 I did not join them. The paper should have used 960h, but I only used 100h now. (At first, I just want to be faster. I will increase it to 960h to keep the same after confirming that the configuration and the machine are ok.)

If you use sets: ['train-clean-100', 'train-clean-360', 'train-other-500'],
then use total_steps: 1000000 for the model to converge, which is approximately 25 days.
If sets: [train-clean-360'], then total_steps: 500000 is enough (empirically).
If sets: [train-clean-100'], then total_steps: 200000 is enough (empirically), which is approximately 5 days.

200000

I got it!
My config on TERA now is
TERA
single GPU, 16/24G
total_steps: 1000000,gradient_accumulate_steps: 4, train_batch_size: 8, sets: ['train-clean-100']
{"@timestamp":"2022-06-16 10:02:51.304","@message":"\roverall:   0%|          | 0/1000000 [00:00<?, ?it/s]"}
{"@timestamp":"2022-06-17 09:31:52.716","@message":"\rtrain:  70%|███████   | 2495/3564 [20:47<09:14,  1.93it/s]\u001b[A\roverall:   4%|▍         | 42501/1000000 [23:29:01<532:30:39,  2.00s/it]"}

23h31min=1411min 42501 steps. At this rate, I seem to need 25 days. I should set total_steps: 200000 if I only use sets: ['train-clean-100'] on a single GPU.
By the way, based on your experience, how to set gradient_accumulate_steps=? and train_batch_size=? when using 4GPU or 8GPU or more on pretain model trainning?

Moreover, I also test on sv task based on wavlm_large.
python run_downstream.py -m train -u wavlm_large -d sv_voxceleb1 -n wavlm_large_single
python -m torch.distributed.launch --nproc_per_node 4 run_downstream.py -m train -n wavlm_large_ddp4 -u wavlm_large -d sv_voxceleb1 \     -o config.runner.gradient_accumulate_steps=2
Wavlm_large for sv_voxceleb1
single GPU， 4.5/24G
config default: gradient_accumulate_steps: 5, train_batch_size: 10
{"@timestamp":"2022-06-17 00:13:25.409","@message":"\roverall:   0%|          | 0/200000 [00:00<?, ?it/s]"}
{"@timestamp":"2022-06-17 09:30:20.557","@message":"\rtrain:  15%|█▍        | 2262/15349 [26:04<2:30:33,  1.45it/s]\u001b[A\roverall:   5%|▍         | 9662/200000 [9:16:55<182:44:30,  3.46s/it]"}
9h17min=557min 9662 steps

----
Wavlm_large for sv_voxceleb1
single GPU， 22/24G
config change: gradient_accumulate_steps: 2, train_batch_size: 80
{"@timestamp":"2022-06-16 19:38:26.416","@message":"\roverall:   0%|          | 0/200000 [00:00<?, ?it/s]"}
{"@timestamp":"2022-06-17 09:31:22.989","@message":"\rtrain:  60%|█████▉    | 1145/1919 [1:48:14<1:13:07,  5.67s/it]\u001b[A\roverall:   2%|▏         | 4411/200000 [13:52:56<618:46:48, 11.39s/it]"}
13h43min=823min 4411 steps

----
Wavlm_large for sv_voxceleb1
4 GPU DDP, each 22/24G
config change: gradient_accumulate_steps: 2, train_batch_size: 80
{"@timestamp":"2022-06-16 19:42:41.434","@message":"\roverall:   0%|          | 0/200000 [00:00<?, ?it/s]"}
{"@timestamp":"2022-06-17 09:28:43.286","@message":"\rtrain:   1%|          | 3/480 [00:24<58:37,  7.37s/it]  \u001b[A\roverall:   2%|▏         | 4322/200000 [13:46:02<694:23:39, 12.78s/it]"}
13h46min=826min 4322 steps

We can see that:

compare single GPU case, one is default gradient_accumulate_steps: 5, train_batch_size: 10,  it reached 9662 steps in less time; The other one is gradient_accumulate_steps: 2, train_batch_size: 80. It only reached 4411 steps and took more time. In here, The larger the bs, the slower the training speed.
conpare single GPU with 4 GPU DDP, they are same gradient_accumulate_steps: 2, train_batch_size: 80 and the also take nearly the same number of steps in the same amount of time. Which means under this config that there is no speed increase for multi-GPU training.

Conclusions:

Turning up the BS to fill up the video memory GPU does not play a role in speeding up;
Using the default configuration of s3prl, although it only takes up a small part of the GPU memory, it is faster.

I suddenly remembered that there is Batch Normalization in visual tasks. The larger the BS, the better the features can be learned and the overfitting can be avoided.
Speech is a sequence model. Due to the different lengths, there is no Batch Normalization in the sequence model. The larger the BS, the less frequently the parameters are updated, and the lack of BN for normalization will make the model more difficult to learn. Therefore, the utilization of GPU memory should not be pursued blindly. After all, the larger the BS, the slower the speed, and the effect may be worse.
Again, based on your experience, how to set gradient_accumulate_steps=? and train_batch_size=? when using 4GPU or 8GPU or more on this ASV task?

We can see that:

compare single GPU case, one is default gradient_accumulate_steps: 5, train_batch_size: 10,  it reached 9662 steps in less time; The other one is gradient_accumulate_steps: 2, train_batch_size: 80. It only reached 4411 steps and took more time. In here, The larger the bs, the slower the training speed.
compare single GPU with 4 GPU DDP, they are same gradient_accumulate_steps: 2, train_batch_size: 80 and the also take nearly the same number of steps in the same amount of time. Which means under this config that there is no speed increase for multi-GPU training.

Conclusions:

Turning up the BS to fill up the video memory GPU does not play a role in speeding up;
Using the default configuration of s3prl, although it only takes up a small part of the GPU memory, it is faster.

@leo19941227 Can you comment on this?
Why is gradient_accumulate_steps: 2, train_batch_size: 80 slower than gradient_accumulate_steps: 5, train_batch_size: 10?
Is it the communication cost between devices that are slowing the process down?
Or maybe the cost to extract features on-the-fly for a large batch size is too high?
I believe you are more experienced with DDP than I am.

I suddenly remembered that there is Batch Normalization in visual tasks. The larger the BS, the better the features can be learned and the overfitting can be avoided. Speech is a sequence model. Due to the different lengths, there is no Batch Normalization in the sequence model. The larger the BS, the less frequently the parameters are updated, and the lack of BN for normalization will make the model more difficult to learn. Therefore, the utilization of GPU memory should not be pursued blindly. After all, the larger the BS, the slower the speed, and the effect may be worse.

I believe we can also do Batch Normalization on sequential data.

Again, based on your experience, how to set gradient_accumulate_steps=? and train_batch_size=? when using 4GPU or 8GPU or more on this ASV task? (make training faster)

Maybe Leo could comment on this one as well!

Since it is important to consider the I/O and GPU utility balance when training your models.
In the first config, you must first load 80 data points before feed them into GPU, hence the GPU waits so long (utility 0) before all the 80 data points are loaded.
In the second config, you feed the data points for GPU as soon as 10 samples are loaded, during the GPU forward, you continue to load more data points, hence now both GPU and I/O have proper utility, hence it is faster.
In the first config, you are wasting GPU power most of the time due to I/O bottleneck.
The first config can ideally faster than the second config when all the system resources (I/O, GPU, CPU) can get 100% utility all the time, however this put huge pressure on disk I/O speed and is only possible when you are using SSD.

Hi @andi611, I followed your advice and trained TERA with the default configuration.
On SUPERB: TERA for SV the ERR is 15.89.
I use officeal tera_100hr to finetune downstream task of SV: python run_downstream.py -m train -u tera_100hr -d sv_voxceleb1 -n tera_100hr_G1_5_10, the ERR is 29.83. The result is slower than on SUPERB.
I guess the reason is that I used tera_100hr as base model to finetune SV rather use tera as base model.
Then I train TERA from scratch with the default configuration(only change sets to train_clean_100h), using TERA trained for 350K steps as the base model finetune SV. The ERR is 9.45, The result is higher than on SUPERB.
I guess offical tera_100hr model only trained 200K steps, but I train 350K steps.
Is my understanding correct?


Wrong linkedin profile

Linkedin profile referenced on your streamlit apps redirects to a wrong profile, https://www.linkedin.com/in/qiushengwu/ rather than your https://www.linkedin.com/in/giswqs

Thank you for letting me know. I changed my LinkedIn handle a while ago. Some werid people reuse my handle. Not sure why


animation not working with 1.21.4

Description
animation is not working while using 1.21.4
Reproduction
all set to time and auto play enabled and i added the model to the potion.json and its only happen while using the time behivor
Screenshots
https://streamable.com/vgm8j6
Environment:

Windows 11
1.21.4
RTX 4080 Super / i9-14900KF
https://prnt.sc/N6tabcPBgZ3Q

Additional context
its only happen while using time behivor and its only happen while using minecraft version 1.21.4 with older version its working great

Hi, I would also like to confirm the issue. I am currently not able to select a custom frame using potion color, either.
When using time as any color behavior, it also does not allow autoplay to work.
For the tag I'm using: item:{id:"minecraft:potion",count:1,components:{"minecraft:custom_model_data":{strings:["30101"]},"minecraft:potion_contents":{custom_color:8388608}} where custom color is any value for paused frame to show
Minecraft version: 1.21.4, Latest objmc version


Project status?

Hi! What's the status on this project? There are quite a few TODOs mentioned, and not much activity lately. Is the implementation complete enough, or is it lack of time/interest from community?

I use it all the time in production settings and it's solid for what it's doing. The TODOs in NOTES.org are mostly brain dump content - some might even be good ideas. I just haven't had the cycles to push things beyond what they are now.
Some people use e2 but not enough who are motivated to contribute. I think the Elixir movement has inspired a bit more action on that front :)

Ok, thanks. I think that these more streamlined server behaviours would help beginners understand faster the Erlang way of thinking. The step from raw Erlang to gen_server can be a bit daunting. Maybe with some marketing there will be more interest? Have you tried to push the ideas to the OTP team, besides the "rename OTP" discussion in 2014?


[mysql-connector] Vulnerability CVE-2018-3258

CVE-2018-3258 is applicable to jdbc-mysql 5.1.47 due to the use of mysql-connector-java-8.0.11.jar.  To resolve this issue a version of the connector greater than 8.0.12 needs to be used.  Pull request #1026 resolves this issue by updating the adapter to version 8.0.17.

Possible to allow the fix?  Pull request #1026 resolves this issue by updating the adapter to version 8.0.17.

Cool, thanks for this folks. Sorry it was not seen until now, we'll get a release out soon.

Looks like #1026 has been merged and released so I think we're good to go!


docker/ci: create-docker-image.sh remove bashism

Running under dash generates the error:
scripts/ci/create-docker-image.sh: 20: scripts/ci/create-docker-image.sh: [[: not found
Replaced as the pattern // doesn't match the D1 case is left unchanged:
$ D1=skydive/skydive
$ D2:8080=registry.ng.bluemix.net/skydive/skydive
$ echo ${D1%//}
skydive/skydive
$ echo ${D2%//}
registry.ng.bluemix.net:8080

run skydive-create-docker-image


Why core data storage space is not released？

I create some data then delete them all
and I check out the usage in settings
storage space is not released

The database never gets deleted. If you want to delete the database, you would have to do it manually. You can get the URL by doing:
SwiftRecord.sharedRecord.sqliteStoreURL
Use it with NSFileManager.defaultFileManager().removeItemAtPath


[chip,dv] update include path for pinmux reg from ip to top

This PR address #14198

Signed-off-by: Jaedon Kim<EMAIL_ADDRESS>

@jdonjdon you'll probably need to rebase, there are some CI issues flying around.