Container mountpoint lost after reboot

Hi guys !
I'm currently playing with convoy and managed to use it inside a Swarm cluster using a NFS server to share my volumes between the hosts.
The only thing is that when I restart a host, convoy seems to lost the mountpoint data for the volumes.
Before reboot of the host:
$ sudo convoy list
{
	"b6139d73-666c-4c52-aa6c-7ef912c6a051": {
		"UUID": "b6139d73-666c-4c52-aa6c-7ef912c6a051",
		"Name": "shared-vol",
		"Driver": "vfs",
		"MountPoint": "/var/nfs/volumes/shared-vol",
		"CreatedTime": "Fri May 06 04:26:12 +0000 2016",
		"DriverInfo": {
			"Driver": "vfs",
			"MountPoint": "/var/nfs/volumes/shared-vol",
			"Path": "/var/nfs/volumes/shared-vol",
			"PrepareForVM": "false",
			"Size": "0"
		},
		"Snapshots": {}
	}
}

After reboot:
sudo convoy list
{
	"b6139d73-666c-4c52-aa6c-7ef912c6a051": {
		"UUID": "b6139d73-666c-4c52-aa6c-7ef912c6a051",
		"Name": "shared-vol",
		"Driver": "vfs",
		"MountPoint": "",
		"CreatedTime": "Fri May 06 04:26:12 +0000 2016",
		"DriverInfo": {
			"Driver": "vfs",
			"MountPoint": "",
			"Path": "/var/nfs/volumes/shared-vol",
			"PrepareForVM": "false",
			"Size": "0"
		},
		"Snapshots": {}
	}
}

After restarting the host, if I want to restart a container using a shared volume, I get the following error:
$ docker start nginx1                                                                                Error response from daemon: Error response from daemon: get shared-vol: VolumeDriver.Get: Handler not found: POST /VolumeDriver.Get

Hi @deviantony
Please give a try https://github.com/rancher/convoy/releases/tag/v0.5.0-rc1 . It should fix this issue.
Notice it's not compatible with previous Convoy's configuration file. You may need a new setup to try it on.

In fact it's a duplicate of #112 , will close this one for now.


InfrastructureManager::infrastructureConstructionHandler does not remove demand-triggered infrastructure

Lines 388-401 of InfrastructureManager.cpp on master branch will not properly remove a built project from the demand_infra_construction_order vector if any ROF-triggered projects exist (i.e. if rof_infra_construction_order is not empty) because of the if-elseif structure. Given increasing demand, this can result in repeated implementation of the same demand-triggered project. A solution would be replacing these lines with the following:
                bool rof_check = false; bool demand_check = false;
                if (!rof_infra_construction_order.empty()) {
                    Utils::removeIntFromVector(rof_infra_construction_order, wss);
                    rof_check = true;
                }
                if (!demand_infra_construction_order.empty()) {
                    Utils::removeIntFromVector(demand_infra_construction_order, wss);
                    demand_check = true;
                }
                // if a project was triggered, but removed from neither list, throw error
                if (!rof_check && !demand_check)
                    throw logic_error("Infrastructure option whose construction was"
                                        " complete is not in the demand or "
                                        "rof triggered construction lists.");

Hi Dave, thanks for submitting the issue and sorry about the awfully delayed reply. I've been busy for a while helping a Brazilian MS student using WaterPaths and her schedule is quite tight, which is why I haven't gotten into this yet.
If you have the time and the disposition, would you mind creating a unit test for this and submitting a pull request with your bug fix?


Question about quantization method?

Hi, thank you for sharing the code.
I was wondering whether the quantization method used waas "per-channel" or "per-layer".
I am having trouble looking up the setup in your paper.
Thank you in advance.
Best regards.

Hi there,
We used a per-layer quantization method. There is a single scaling factor for each layer.

Thank you for the reply :)


support more than a fixed number of command line arguments

After removing alloca usage, we now have a hard limit of (arbitrarily chosen) 32 command line arguments. More than this will crash startup. The problem is that we get args in the form of &&u8 and we want to discourage null terminated things, and encourage using slices or situations where the length is known.
There are 2 proposals to solve this:
Proposal 1: use an API to access command line arguments
A simple program to print all the arguments then becomes:
const std = @import("std");
const io = std.io;
const Args = std.os.Args;

pub fn main(args: &Args) -> %void {
    var i: usize = 0;
    while (i < args.count(); i += 1) {
        const arg: []const u8 = args.get(i);
        %%io.stdout.printf("{}={}\n", i, arg);
    }
}

Alternately, this can be absent from the main function, and command line arguments can be accessed via the standard library directly:
const std = @import("std");
const io = std.io;
const os = std.os;

pub fn main() -> %void {
    var i: usize = 0;
    while (i < os.argv.count(); i += 1) {
        const arg: []const u8 = os.argv.get(i);
        %%io.stdout.printf("{}={}\n", i, arg);
    }
}

The latter has the benefit that if args are unused, there is no startup cost in code size or processing. In release mode there should really be no code generated to deal with the arguments if unused.
Proposal 2: use a combination of fixed size buffer and mmap if the count is too large
With this we can retain the current API for main function. If the number of command line arguments is less than some fixed size (32 currently) then we can use the fixed size buffer. Otherwise we request the necessary heap allocation before calling main.
comparison
Proposal 1 would allow a program that did not look at command line arguments to not generate code to deal with them. Meanwhile proposal 2 would always do a mmap if > N number of args were provided, even if those args are ultimately ignored.
Proposal 1 wastes less memory in the case of a large number of command line args - 8 bytes per argument.
Proposal 1 is easier to implement for every OS, whereas in proposal 2 you have to add specific memory allocation code for every OS. mmap on Posix, HeapAllocate on Windows, etc.
Proposal 1 avoids memory allocation, which some programs may want to do. However, in favor of proposal 2, we could make the max number of command line arguments a compile time variable, and set a flag if more args than this are provided.
Proposal 2 API is cleaner. You can do a for loop for example. It will be more familiar to C programmers who expect command line arguments to be passed to main. And it discourages global access of the arguments since they're only available in main and you'd have to capture them to get access to them elsewhere in the program, such as a third party library.
Proposal 2 API lets you do something like the following, that passes leftover args to a child process:
pub fn main(args: [][]u8) -> %void {    const zig_exe = args[1];
    const leftover_args = args[2...];

    os.ChildProcess.spawn("/usr/bin/foo", leftover_args, ...);
}

my understanding is that the main(args: &Args) strategy would be O(1) before main, O(1) for args.count(), and O(1) for args.at(i), where O(1) means userspace runtime is not dependent on the number of command line args given to the program. Is all that true for all platforms?

It's at least true for all platforms when linking libc, since we get argc: c_int and argv: &&u8 from libc main. Probably true for all platforms always.


Lens stops working after 30 minutes in the program.

When using the lens pack which includes all of the popular and more recent lenses, and activating it by searching the link within the program, it loads and works flawlessly for 30 minutes. After this it will suddenly drop out, losing the searcehd lens and needs to be reactivated. The searched lens also disappears from the search results.
Why is SnapCam losing the lens exactly every 30 minutes? Is it pinging docker every 30 mins when no lens change is made?
I have tried activating and deactivating the lens and it does not reset the 30 minute timer. Only removing the lens link in the search window, re searching and reactivating will restart the process.

My SnapCamera app doesn't support URL searching nor do I use Docker. You might be mistaken about what project you're using.

Apologies my bad, this was the starting point for me but I'm running in docker to access new lenses.


LZ Compress Doesn't Actually Compress

I have a save system, and am improving it to make it better compressed with the new Zip extension
Old Save Size: ~9 Megabytes
New Save Size: ~4.77 Megabytes
New (LZ Compress Raw): 5.63 Megabytes
New (LZ Compress Base64): 7.72 Megabytes
New (LZ Compress EncodedURLComponent): 7.72 Megabytes
and some are up to 17 Megabytes
my point is, this is actually Largening the size than compressing it like it should.
please fix this!

Usually, compressing compressed data (since zips are also compressed) doesn't yield much better results, or may increase the size of the data.

Okay, so it's basically intended behaviour?

Yeah. Sb3 files are already compressed and compressing compressed data doesn't really help.
The extension might be useful to combine with eg. Local storage or other places that aren't compressed


Error when trying to run sample code

If im trying to run your sample code I always get this error:
C:\Users\PC-Werkstatt\Documents\Arduino\libraries\TCA9548A\src\TCA9548A.cpp: In member function 'void TCA9548A::begin(uint8_t, uint8_t, TwoWire&)':

C:\Users\PC-Werkstatt\Documents\Arduino\libraries\TCA9548A\src\TCA9548A.cpp:18:34: error: no matching function for call to 'TwoWire::begin(uint8_t&, uint8_t&)'

 this->myWire->begin(_sda,_sdl);

In file included from C:\Users\PC-Werkstatt\Documents\Arduino\libraries\TCA9548A\src\TCA9548A.h:13:0,

from C:\Users\PC-Werkstatt\Documents\Arduino\libraries\TCA9548A\src\TCA9548A.cpp:9:

C:\Program Files (x86)\Arduino\hardware\arduino\avr\libraries\Wire\src/Wire.h:52:10: note: candidate: void TwoWire::begin()

void begin();

im using a arduino nano with the following pins:

#include "TCA9548A.h"
#define SCLPIN 5    // Your I2C pins
#define SDAPIN 4    // Your I2C pins

Do you have any idea?

Hi Stephan, sorry I missed this over a month ago.
I had only tested this library with the ESP32. Turns out, the Arduino Uno begin only accepts an address (for acting in slave mode), so adding I2C pins makes it throw an error.
I think is due to the Uno having only one set of I2C pins, so it assumes there's no need to add the additional argument.
It should be working now - though I imagine you've made past this library some time ago.
Cheers for the raising the issue,


Tabbed Navigation loses scroll position

The tab screen is unmounted on Switching between tabs and it re-render every time. If i add scrollview inside color tab and scroll to bottom. On switching tab scroll position is gone.
What should i change in Navigation Experiment to achieve this?

+1

@nak1b http://stackoverflow.com/a/39134505

Having the same issue. When switching tabs, all the views in that tab are unmounted. When switching back to the tab, everything is remounted again. How can I avoid this happening?

@zachrnolan If you don't want tabs to be mount/unmount every time you switch tab, render all of them in render of navigationView

there is a question about it ; http://stackoverflow.com/questions/39007521/navigation-experimental-is-unmounting-tabs
I think the way navigation stack handling wrong here switching tabs shouldn't be unmounting


RipMe won't start anymore since latest Java update

Ripme version: ripme-2.1.9-7-22e915df.jar
Java version: java version "22.0.1" 2024-04-16
Java(TM) SE Runtime Environment (build 22.0.1+8-16)
Java HotSpot(TM) 64-Bit Server VM (build 22.0.1+8-16, mixed mode, sharing)
Operating system:  Windows 11: Microsoft Windows [Version 10.0.22631.4037]

Exact URL you were trying to rip when the problem occurred: n/a
Please include any additional information about how to reproduce the problem: n/a

Expected Behavior
Starting RipMe
Actual Behavior
Since the latest Java update RipMe won't start anymore, it gives the error:
Title: Java Virtual Machine Launcher
Error message: A Java Exception has occurred.

Try installing Java SE Development Kit 17.0.11 seems to fix it for me.

Thanks, it worked! :)

Gesendet: Donnerstag, 29. August 2024 um 21:21 Uhr
Von: "CosmicKahos" @.>
An: "RipMeApp/ripme" @.>
Cc: "KTriebler" @.>, "Author" @.>
Betreff: Re: [RipMeApp/ripme] RipMe won't start anymore since latest Java update (Issue #2046)

Try installing Java SE Development Kit 17.0.11 seems to fix it for me.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>


cargo doesn't know which to use because multiple target files found

First of all thanks for providing the flake! It's really awesome and simplifies packaging rust projects for nix.
Describe the bug
Starting with commit 62e50137688d953557f156f01e2ad2a25b22d66c, I cannot build my projects anymore, that only contain binaries within the src/bin/ directory.
I'm using this flake in https://github.com/dj95/zjstatus to package the project. Simplified, the project setup is
src/bin/zjstatus.rs
src/bin/zjframes.rs
src/lib.rs
flake.nix

When building the flake of zjstatus with nix flake check . when using a commit newer than 62e50137688d953557f156f01e2ad2a25b22d66c, the build will fail with the following error:
       > Running phase: configurePhase
       > will append /private/tmp/nix-build-zjstatus-deps-0.19.1.drv-0/source/.cargo-home/config.toml with contents of /nix/store/8fyaknmzzlgzmfxzn0my7fjwrcm3sykb-vendor-cargo-deps/config.toml
       > default configurePhase, nothing to do
       > Running phase: buildPhase
       > ++ command cargo --version
       > cargo 1.83.0 (5ffbef321 2024-10-29)
       > ++ command cargo check --release --target wasm32-wasip1
       > error: failed to parse manifest at `/private/tmp/nix-build-zjstatus-deps-0.19.1.drv-0/source/Cargo.toml`
       >
       > Caused by:
       >   cannot infer path for `zjstatus` bin
       >   Cargo doesn't know which to use because multiple target files found at `src/main.rs` and `src/bin/zjstatus.rs`.
       For full logs, run 'nix log /nix/store/vmc65y5xzzpiv36gbb6qz4b0fix9dsvx-zjstatus-deps-0.19.1.drv'.

When using a commit before the specific one, everything works fine.
This also occurs in different other projects I maintain.
Reproduction

git clone https://github.com/dj95/zjstatus
Apply the following patch with git apply PATCH_FILE_NAME

diff --git a/flake.lock b/flake.lock
index 5185d1b..a27e0ec 100644
--- a/flake.lock
+++ b/flake.lock
@@ -2,17 +2,17 @@
   "nodes": {
     "crane": {
       "locked": {
-        "lastModified":<PHONE_NUMBER>,
-        "narHash": "sha256-mlIDSv1/jqWnH8JTiOV7GMUNPCXL25+6jmD+7hdxx5o=",
+        "lastModified":<PHONE_NUMBER>,
+        "narHash": "sha256-0fJaoI4B9Nn67E1P44usZhZHkSSyWdAI23HU+X+HJCQ=",
         "owner": "ipetkov",
         "repo": "crane",
-        "rev": "af1556ecda8bcf305820f68ec2f9d77b41d9cc80",
+        "rev": "62e50137688d953557f156f01e2ad2a25b22d66c",
         "type": "github"
       },
       "original": {
         "owner": "ipetkov",
         "repo": "crane",
-        "rev": "af1556ecda8bcf305820f68ec2f9d77b41d9cc80",
+        "rev": "62e50137688d953557f156f01e2ad2a25b22d66c",
         "type": "github"
       }
     },
diff --git a/flake.nix b/flake.nix
index e96747d..b834e7a 100644
--- a/flake.nix
+++ b/flake.nix
@@ -5,7 +5,7 @@
     nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

     crane = {
-      url = "github:ipetkov/crane/af1556ecda8bcf305820f68ec2f9d77b41d9cc80";
+      url = "github:ipetkov/crane/62e50137688d953557f156f01e2ad2a25b22d66c";
     };

     flake-utils.url = "github:numtide/flake-utils";

Run nix flake check . within the patched repository

Feel free to message me in case you need more information.

Works like a charm. Thanks a lot for the quick fix!


Add pagination to [GET] /workspaces if we expect to have a large amount of workspaces

[GET] /workspaces could potentially return a lot of workspaces, suggestion from @sachinkundu to add pagination to the API

I am retiring this issue - we would have to have a lot (100s) of concurrent workspaces for this to make sense and that is not likely


add reg method

double click cursorOpenFolder.bat, it will generate a temp.reg to register
add three kind menu:

for file: edit with cursor
for foder
for blank space in foder explorer

reference:
windows 7 - Visual Studio Code "Open With Code" does not appear after right-clicking a folder - Stack Overflow

Thanks! :) Could you explain why you prefer using bat and reg instead of PowerShell? Is it a matter of preference, or is PowerShell not installed?

Thanks! :) Could you explain why you prefer using bat and reg instead of PowerShell? Is it a matter of preference, or is PowerShell not installed?

:) Thanks In fact, I don't understand the reg method either. It's because the PowerShell script didn't run successfully on my laptop. So searched for a solution for vscode on the Internet. Then the other party used the reg method. After experimenting, I found it was successful, so I added it.

Thanks @eatcosmos; it worked very well!


Messages refactor

WIP

This is pretty done. Maybe I should move hooks to separate files?
This PR just removes redux, after I will work on messages further.

This is pretty done. Maybe I should move hooks to separate files?
This PR just removes redux, after I will work on messages further.

Build is choking on FriendRequestsReceived.stories.tsx I've done since they're still referring the Redux store in the story file...

Build is choking on FriendRequestsReceived.stories.tsx I've done since they're still referring the Redux store in the story file...


cant unadmin console error

I was trying to unadmin myself to check default permissions but i get this error.
[3/21/2016 1:39:50 AM] [Info] Successfully unadmined arjetiko
[3/21/2016 1:39:50 AM] [Error] An error occured while executing unadmin [arjetiko]: System.ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
at System.Collections.Generic.List1[SDG.Unturned.SteamPlayer].get_Item (Int32 index) [0x00000] in <filename unknown>:0    at SDG.Unturned.SteamAdminlist.unadmin (CSteamID playerID) [0x00000] in <filename unknown>:0    at Rocket.Unturned.Player.UnturnedPlayer.Admin (Boolean admin, Rocket.Unturned.Player.UnturnedPlayer issuer) [0x00000] in <filename unknown>:0    at Rocket.Unturned.Player.UnturnedPlayer.Admin (Boolean admin) [0x00000] in <filename unknown>:0    at Rocket.Unturned.Commands.CommandUnadmin.Execute (IRocketPlayer caller, System.String[] command) [0x00000] in <filename unknown>:0    at Rocket.Core.Commands.RocketCommandManager+RegisteredRocketCommand.Execute (IRocketPlayer caller, System.String[] command) [0x00000] in <filename unknown>:0    at Rocket.Core.Commands.RocketCommandManager.Execute (IRocketPlayer player, System.String command) [0x00000] in <filename unknown>:0
Also i never could unadmin a player that isnt in the server.
unadmin<PHONE_NUMBER>9190355
Invalid parameter
Thank you

Are you running the duty plugin?
Remove your admins.dat file [b]or[/b] the line that says "76561198179190355".

I deleted adminlist.dat restarted the server and it works thank you!!

fr34ky should give me permissions to close an issue.. lol.

That might be the source of the problem when you install it and already have administrators assigned in admins.dat.


Added some properties for image

In some cases properties "og:image:secure_url" and "thumbnailUrl" are also mandatory.

@troxler I need this in your lib, pls :)

@troxler ??

@kaskar2008 I will look at it as soon as possible. Until then, you could use the head property.

Do you have a usecase or a reference describing when you should use

property="og:image" and property="og:image:secure_url" with the same URL? As far as I understand from the documentation, the latter only has to be specified when the former is not an HTTPS URL. So setting both the same HTTPS URL seems useless.
itemprop="image" and itemprop="thumbnailUrl" with the same URL? They seem to have different semantics and setting both to the same URL seems overkill. Do you know any service that only reads itemprop="thumbnailUrl" and ignores itemprop="image"?

Closing due to inactivity.


Outgoing lan rules with specific gateway not functional

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
Outgoing rules from lan with a specific gateway no longer work on version 24.7_9.
To Reproduce
Multiple wan/gateway needed
Steps to reproduce the behavior:
Need multi-WAN with 2 gateway
Creates an X alias of type URL (ips) set to "my-ip.com" content
Creates a firewall rule on the lan interface that states.

Action: Pass
interface: LAN
Direction: out
TCP/IP Version: IPV4
Protocal: any
Source: LAN net
Destination: alias X
Gateway: WAN2

traceroute my-ip.com to validate that the rule is not applied
Software version used and hardware type if relevant, e.g.:
OPNsense 24.7_9 (amd64).

I believe Direction: in is the proper way. Setting this to out would have this rule applied after routing took place.

This rule worked very well in 23.7 since the switch to 24.x this no longer works.
I need it to go out on a specific wan, not come in on a specific wan. Otherwise, I specify the source as PC-XXX direction: out and it works, but I prefer to manage the output via url alias.

Quite the broad statement: 23.7 since the switch to 24.x. Are you sure this didn't work on 24.1.x? Because it would make more sense it did and now on 24.7 it doesn't work (because FreeBSD 14.1 and related work).

I can neither confirm nor deny that it didn't work on 24.1. I do know, however, that it worked on version 23.7.

Fair enough. I have to park this for now unless someone chimes in because I won't go through a year's worth of commits.

After trying it out, it works well in 24.1


InsecureTLSVerify hard-coded to true, exposing MITM risk

The client is subject to man-in-the-middle certificate spoofing, as the TLS checking is hard-coded as disabled here.
Consider driving this option via environment variable such as AQUA_TLS_VERIFY=0, defaulting the option to true and enabling TLS verification by default.

This appears set across all the client requests, including registry requests which send a Basic auth header in the request as well.


Fix typos in source code comments

Critical Changes
None
Changes
Fix a couple of typos in source control. (ff97ca1)
Issues Closed
#397

@ryanguest Thank you for the contribution!


ModernMT does not run on Geforce 30 series - verified with clean installation

@davidecaroselli @nicolabertoldi

Back-up your existing engines so that you can copy them back later.
Reinstall Ubuntu 18.02 from a USB stick. Note that Ubuntu will not recognize your graphics card and use the low-res nouveau drivers instead.
Do not install the newest NVIDIA drivers via the NVIDIA website.
Instead install CUDA 11.1 (the only version compatible with the 455.32 drivers needed for your RTX 3090) straight away via the same website. This installation includes the drivers (455.32). Contrary to 3), these will work on Ubuntu.
Install cuDNN 8.0.4 via the NVIDIA website (the only version compatible with CUDA 11.1).
Follow rest of installation instructions.

Card is detected as GPU 0 in nvidia-smi. No errors anywhere or whatsoever.
Starting engine "engdut".... OK
Loading models...
(The end - nothing ever happens - it gets stuck here)
node.log:
2020-11-04 14:47:55,557 INFO  eu.modernmt.cluster.ClusterNode [main] Starting node in cluster "mmt-4.9.4-202011041334" with config:
Node: load-balancing=true
Network: interface='<IP_ADDRESS>', port=5016, host='<IP_ADDRESS>'
Api: enabled=true, port=8045, root='null'
Join: timeout=5
Binlog: enabled=true, embedded=true, hosts=localhost, port=9092, name='null'
Database: embedded=true, enabled=true, type=CASSANDRA, port=9042, host='localhost', name='null', user='null', password='****'
Engine: name='engdut', languages=1
Decoder: queue=400, threads=5, gpus=0, class='null', enabled=true, echo=false
Aligner: enabled=false
Analyzer: enabled=true, analyze=true, batch=16, threads=4, timeout=30, misalignment=10240
2020-11-04 14:47:55,567 INFO  eu.modernmt.cluster.ClusterNode [main] Node is joining the cluster
2020-11-04 14:47:56,121 INFO  eu.modernmt.cluster.ClusterNode [main] Node joined cluster of 1 members in 0.4s
2020-11-04 14:47:56,200 INFO  eu.modernmt.cluster.ClusterNode [main] Model loading started
2020-11-04 14:47:56,201 INFO  eu.modernmt.engine.Engine [main] Loading default pre-processing configuration
2020-11-04 14:47:56,211 INFO  eu.modernmt.engine.Engine [main] Loading default post-processing configuration
2020-11-04 14:47:56,419 INFO  eu.modernmt.decoder.neural.queue.DecoderQueueImpl [pool-6-thread-1] Native decoder process on GPU 0 is starting
2020-11-04 14:51:01,406 INFO  eu.modernmt.cluster.ClusterNode [ShutdownThread] Halting API interface...
2020-11-04 14:51:01,411 INFO  eu.modernmt.cluster.ClusterNode [ShutdownThread] API interface halted
2020-11-04 14:51:01,411 FATAL eu.modernmt.facade.ModernMT [ShutdownThread] Unexpected exception thrown by thread [ShutdownThread]
java.lang.IllegalStateException: ClusterNode not ready. Call bootstrap() to initialize the member.
at eu.modernmt.cluster.ClusterNode.getEngine(ClusterNode.java:96)
at eu.modernmt.cluster.ShutdownThread.run(ShutdownThread.java:36)
(The latter is because I pressed Ctrl+C)
gc.log:
OpenJDK 64-Bit Server VM (25.272-b10) for linux-amd64 JRE (1.8.0_272-8u272-b10-0ubuntu1~18.04-b10), built on Oct 26 2020 02:23:20 by "buildd" with gcc 7.5.0
Memory: 4k page, physical 65823628k(19959848k free), swap 2097148k(2097148k free)
CommandLine flags: -XX:+CMSClassUnloadingEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:ErrorFile=/home/loek/modernmt/runtime/engdut/logs/hs_err_pid%p.log -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/loek/modernmt/runtime/engdut/logs -XX:InitialHeapSize=16106127360 -XX:MaxHeapSize=16106127360 -XX:MaxNewSize=697933824 -XX:MaxTenuringThreshold=6 -XX:NewSize=697933824 -XX:OldPLABSize=16 -XX:OldSize=1395867648 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+ScavengeBeforeFullGC -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2020-11-04T15:36:26.452+0100: 2,213: [GC (CMS Initial Mark) [1 CMS-initial-mark: 0K(15047104K)] 403559K(15660544K), 0,0240153 secs] [Times: user=0,03 sys=0,00, real=0,02 secs]
2020-11-04T15:36:26.477+0100: 2,237: [CMS-concurrent-mark-start]
2020-11-04T15:36:26.481+0100: 2,242: [CMS-concurrent-mark: 0,005/0,005 secs] [Times: user=0,01 sys=0,00, real=0,01 secs]
2020-11-04T15:36:26.481+0100: 2,242: [CMS-concurrent-preclean-start]
2020-11-04T15:36:26.495+0100: 2,256: [CMS-concurrent-preclean: 0,014/0,014 secs] [Times: user=0,01 sys=0,00, real=0,01 secs]
2020-11-04T15:36:26.495+0100: 2,256: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2020-11-04T15:36:31.561+0100: 7,322: [CMS-concurrent-abortable-preclean: 0,571/5,066 secs] [Times: user=0,60 sys=0,00, real=5,07 secs]
2020-11-04T15:36:31.562+0100: 7,322: [GC (CMS Final Remark) [YG occupancy: 425373 K (613440 K)]2020-11-04T15:36:31.562+0100: 7,322: [GC (CMS Final Remark) 2020-11-04T15:36:31.562+0100: 7,322: [ParNew: 425373K->30271K(613440K), 0,0493859 secs] 425373K->30271K(15660544K), 0,0494610 secs] [Times: user=0,24 sys=0,01, real=0,05 secs]
2020-11-04T15:36:31.611+0100: 7,372: [Rescan (parallel) , 0,0055824 secs]2020-11-04T15:36:31.617+0100: 7,377: [weak refs processing, 0,0000218 secs]2020-11-04T15:36:31.617+0100: 7,377: [class unloading, 0,0031353 secs]2020-11-04T15:36:31.620+0100: 7,381: [scrub symbol table, 0,0034095 secs]2020-11-04T15:36:31.623+0100: 7,384: [scrub string table, 0,0002621 secs][1 CMS-remark: 0K(15047104K)] 30271K(15660544K), 0,0623959 secs] [Times: user=0,29 sys=0,01, real=0,06 secs]
2020-11-04T15:36:31.624+0100: 7,385: [CMS-concurrent-sweep-start]
2020-11-04T15:36:31.624+0100: 7,385: [CMS-concurrent-sweep: 0,000/0,000 secs] [Times: user=0,00 sys=0,00, real=0,00 secs]
2020-11-04T15:36:31.624+0100: 7,385: [CMS-concurrent-reset-start]
2020-11-04T15:36:31.707+0100: 7,467: [CMS-concurrent-reset: 0,083/0,083 secs] [Times: user=0,04 sys=0,07, real=0,08 secs]
While loading, it does create a node.pid and a node.properties in the correct runtime directory.
When I check the status of said engine using ./mmt status, it says it's running, but when actually accessing it via the REST API, I get: {"status":500,"error":{"type":"IllegalStateException","message":"ClusterNode not ready. Call bootstrap() to initialize the member."}}

@davidecaroselli @nicolabertoldi Can you at least let me know when this issue can be solved? Else I will need to take out the 3090 again and re-install the 2080, as this is a production machine which really needs to work before tomorrow morning again. Thank you in advance!

Okay, it was worth a try. Removing graphics card, reinserting the old 2080. Please let me know when this issue is solved; I won't be able to test until next Sunday though.

@LoekvanKooten
actually we never tried testing our software on the RTX3-30 series.
So we do not know which is the right configuration you require.
Concerning GEForce 2080 Ti, the requirements and settings reported in the installation guidelines of the latest version work fine.
Ubuntu 18.04
Java 8
Python 3.6 (or higher)
CUDA Toolkit 10.1
libcudnn7_7.6.5
torch==1.3.0
fairseq==0.7.1
please feel free to close the issue it this works for you as well.

Thank you for your reply!. No, that does not work. Definitely not. The 30 series require 455.32 drivers, which require CUDA 11.1, which requires cuDNN 8.0.4.
I'd definitely start testing this, as the speed increase during training can be enormous!
Are there any plans for this?

@LoekvanKooten
we do not have short-time plan to test ModernMT on this kind of GPUs (especially because we not such GPU available) and modified the code to be compliant with CUDA 11.1 and cuDNN 8.0.4 and 455.32 drivers.
We are thinking to move ModernMT to work with more recent version of fairseq, but we cannot say an expected time.
You said "the speed increase during training can be enormous"; could you please report some figures about that? Which is the GPU you compared to?

That would be a 2080 RTX versus a 3090 RTX.
https://gpu.userbenchmark.com/Compare/Nvidia-RTX-3090-vs-Nvidia-RTX-2080-Ti/4081vs4027#:~:text=The 3090 offers more than,find value in the 3090.
Or am I totally wrong?

@nicolabertoldi Please consider doing this. It may be just a few small adjustments in the code, I don't know. It could potentially shorten 3 days training time for a medium engine to 2 days or less, which is considerable, I think.

@LoekvanKooten
I think that a right comparison should be done on a real ModernMT training/translation task. Not always the declared speedups correspond to real gains.
Furthermore, the 3090 has a double speed, but it costs twice. So for the sake of ModernMT training nothing changes, assuming you buy two 2080Ti GPUs and use the same model size.
By the way moving to newer fairseq is in out to-do list.

@nicolabertoldi Not all systems can fit two 2080's ;-) Also, at least on my system, using multiple GPU's made it very unstable. Please refer to https://github.com/modernmt/modernmt/issues/559

That, plus the fact that the 30 series is the new standard from now on and the 20 series is being phased out. Last but not least, lower cards in this series (the 3070 for example) offer twice the performance of a 2080 for half the price! That's a pretty compelling argument, I'd say.

@nicolabertoldi Thank you for your reply!. No, that does not work. Definitely not. The 30 series require 455.32 drivers, which require CUDA 11.1, which requires cuDNN 8.0.4.

I have currently MMT running with a 455.32 driver and CUDA 11.1, but on a 2080Ti (I don't know for 30 series):
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.32.00    Driver Version: 455.32.00    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:09:00.0 Off |                  N/A |
| 72%   66C    P2    60W / 250W |   8823MiB / 11019MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  On   | 00000000:43:00.0 Off |                  N/A |
| 39%   50C    P2    64W / 250W |   5747MiB / 11018MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

It was a progressive upgrade, thus possibly older versions are also installed with their dependences.
You can install older CUDA version over newer ones, this works (see https://github.com/modernmt/modernmt/issues/482).

Yes, with the 2080 Ti this works. I have tried that. Unfortunately, the 30 series changes all that.
As for now, this is how this ends (funny story).
https://www.linkedin.com/feed/update/urn:li:activity:6729815974389919744/
But I'd rather see another solution :)

Yes, with the 2080 Ti this works. I have tried that. Unfortunately, the 30 series changes all that.

Reading your discussion, I'm not sure we are really on the same line. Did you really do this, in this order:

install Nvidia 455.32 drivers with its required CUDA 11.1 and cuDNN 8.0.4
install over it older CUDA 10.1 with cuDNN 7.6.5 as listed in the MMT requirements
install MMT itself

This may work, but the only reason why it may work, is that ModernMT then uses CUDA 10.1, which does not address the Ampere architecture of the 30 series. So in a best-case scenario this would make the 30-series run as a 20-series, offering no speed advantage.
Or am I totally wrong?

I think, if it's really a question of card architecture, it wont' work.
If it works:

it means there is just a question of lib dependences that could be solved by installing CUDA 11 before CUDA 10 (as I did in the above quoted post)
you will get much more CUDA cores with the 3090 than with the 2080, thus much calculation power, especially for the training config
yo also will get much more G-RAM, thus wil lbe able to load more models on a single card in a production config

I did get the card to work during the first try, after installing and reinstalling stuff for like 8 hours at a stretch. So yes, there is a dependency problem somewhere. In the end however, that led to this problem in fairseq: https://github.com/pytorch/fairseq/issues/2842
Loading models was possible, but training always went wrong when trying to set up the decoder.

Ok, it's up to you to take the time for this test or not. I do not have a 3090 on my side.
Playing Modern Warfare with 600 FPS is also nice.
:D

LOL 👍
As said, I tested that. We really need ModernMT to come up with a solution, and somehow get them to understand why it's a really bad idea to ignore the 30 series.

You tested exactly what I suggested? It's not really clear in all your past explanations.

I did that yesterday, but as soon as I installed CUDA 7.5.6, the card went blank and I only got a low-res screen in Ubuntu. X-Server was no longer working and ModernMT saw no GPU.
It's clear that Ubuntu itself isn't completely ready for the 30 series yet either.

Basically I got the best result during the first try, but I have no idea what I was doing there. That was just 8 hours of installing, de-installing and re-installing everything, from drivers to CUDA to cuDNN fo fairseq to Py-torch. And then snap suddenly models were loading. But then it turned out it was merely using the CPU.

@davidecaroselli @nicolabertoldi
I'm still waiting for an answer :) I think I gave you a very compelling argument for supporting the 30 series. Please include support! If you're not convinced yet, please read the below.
https://www.pcgamer.com/nvidia-rtx-3080-review-founders-edition-benchmarks/
"It feels almost redundant to call the new Nvidia RTX 3080 the fastest graphics card you can buy. I mean, today, it absolutely is, and after the first reveal that was pretty much inevitable. The initial performance figures from the green team's grand unveiling were no joke: this thing takes the RTX 2080 Ti outside and gives it a good, sound kicking. It's not even close.
The ol' Turing GPU's now whimpering in the corner, its vapour chambers leaking out over my floor as its fans fitfully spin their last.
Which is a pretty spectacular result when you consider the top-end Turing chip was the previous holder of the 'fastest graphics card' title, with its bold specs list and $1,200 sticker price. The fact this $699 card can make such a performance step up, at the same price as the RTX 2080 Super it's replacing, is incredible.
It's at once testament to the new Nvidia Ampere architecture, the work its engineers have put in, and maybe to the more competitive landscape of 2020."

@EtienneAb3d
Thinking that maybe, somewhere, I made a mistake, I reinstalled Ubuntu, put brand-new NVIDIA drivers on it, then once more did exactly as you stated, exactly in the order you stated.
When I start an engine, it says:
Loading models...
The fans spin up.
The fans spin down.
The fans spin up.
The fans spin down.
The fans spin up.
The fans spin down.
The fans spin up.
The fans spin down.
Et cetera.
But the model is not loaded.
No error message.
It's a fact. The 3090 does not work with ModernMT. It does not.

Ok. Nice to know before buying a 30XX card !
:-/

@davidecaroselli @nicolabertoldi
Are you discontinueing this product? I'm not the first one asking on this forum, but just like others I've noticed that many messages get very late responses, or no response at all. What's the purpose of a forum if the users don't get any response from the devs anyway?
If this is a matter of not having a good enough ROI on this forum, as said before, I'm willing to pay for a license (as long as it's a 3 digit number) if that means getting quicker and better support.
I'd be much obliged!
I once more notice that the multiple GPU set-up I'm forced to use now results in constant system crashes due to timing issues (not power issues). I'm now forced to train on 1 2080 RTX while I have a very beefy 3090 RTX laying right beside me.
Please let us know when you will implement support for the 30 series. As stated above, there are some very compelling arguments (please read them).

Today we get our new machine with 2x RTX 3090... Same problem as mentioned.
I'm going to investigate it more. But it's very sad to see ModernMT ignoring new GPU series.

I will be posting updates about my findings.

RTX 3090 first support was added in NVIDIA driver 455.23.04 (see release)
only CUDA 11.1 is compatible with 455 driver (see compatibility)
official Pytorch binaries doesn't currenly support CUDA 11.1 (see issue)
unofficial Lambda Stack package for Ubuntu 20.04 claims support for CUDA v11.1, PyTorch v1.6.0, cuDNN v7.6.5 (see blog)
ModernMT requires Pytorch 1.3.0 (see requirements)
ModernMT requires tensorboardX 1.6 and fairseq 0.7.1 (see requirements) but both are Pytorch dependent

If we can update ModernMT to support Pytorch 1.6.0, this could bring us a hope.

So far I have done following.

Download Ubuntu 20.04 LTS (link)
Flash USB drive with Ubuntu image using (balenaEtcher)
Install Ubuntu (and fix problem with network)
Update Ubuntu using Software Updater and restart
Install NVIDIA driver sudo apt install nvidia-driver-455 (link) and restart
Install Lambda stack (link)

LAMBDA_REPO=$(mktemp) && \
wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && \
sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && \
sudo apt-get update && sudo apt-get install -y lambda-stack-cuda

Install java sudo apt install -y openjdk-8-jdk
Install development tools sudo apt install -y git maven cmake libboost-all-dev zlib1g-dev libbz2-dev
Clone ModernMT git clone https://github.com/modernmt/modernmt.git modernmt && cd modernmt
Remove torch==1.3.0 line from requirements.txt
Run installation script python3 setup.py
Compile binaries

pushd src
mvn clean install
popd

Compilation was successful. Now the funny part ./mmt create en it examples/data/train/ --train-steps 10000
click to see whole output

=========== TRAINING STARTED ===========
ENGINE:   default
LANGUAGE: en > it
(1/4) Cleaning corpora
(1/6) Cleaning corpora...                                    DONE in 1s
(2/6) De-duplicate corpora...                                DONE in 0s
(3/6) Preprocess corpora...                                  DONE in 0s
(4/6) Creating aligner filter...                             DONE in 0s
(5/6) Scoring corpora...                                     DONE in 0s
(6/6) Apply aligner-based filter...                          DONE in 0s
(1/4) Cleaning corpora DONE in 3s
(2/4) Creating engine
(2/4) Creating engine DONE in 0s
(3/4) Generating binary archives
(1/4) Tokenize corpora...                                    DONE in 1s
(2/4) Training BPE model...                                  DONE in 8s
(3/4) Encoding training corpora...                           DONE in 0s
(4/4) Generating binary data...                              DONE in 0s
(3/4) Generating binary archives DONE in 11s
(4/4) Training neural model
(1/2) Train neural network...
ERROR Unexpected exception: Command 'fairseq-train /home/nmtuser/Documents/modernmt/runtime/default/tmp/training/data_generated --save-dir /home/nmtuser/Documents/modernmt/runtime/default/tmp/training/_temp_train/nn_model --task mmt_translation --user-dir /home/nmtuser/Documents/modernmt/build/mmt-4.9.4.jar/mmt --share-all-embeddings --no-progress-bar --tensorboard-logdir /home/nmtuser/Documents/modernmt/runtime/default/tmp/training/_temp_train/tensorboard_logdir --dataset-impl mmap --max-update 10000 --arch transformer_mmt_base --clip-norm 0.0 --label-smoothing 0.1 --attention-dropout 0.1 --dropout 0.3 --weight-decay 0.0 --criterion label_smoothed_cross_entropy --optimizer adam --adam-betas (0.9, 0.98) --log-interval 100 --lr 0.0005 --lr-scheduler inverse_sqrt --min-lr 1e-09 --warmup-init-lr 1e-07 --warmup-updates 4000 --max-tokens 3072 --update-freq 4 --save-interval-updates 1000 --keep-interval-updates 10 --keep-last-epochs 10' failed with exit code 1
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq_cli/train.py", line 265, in distributed_main
main(args, init_distributed=True)
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq_cli/train.py", line 80, in main
train(args, trainer, task, epoch_itr)
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq_cli/train.py", line 121, in train
log_output = trainer.train_step(samples)
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq/trainer.py", line 283, in train_step
raise e
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq/trainer.py", line 258, in train_step
loss, sample_size, logging_output = self.task.train_step(
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 234, in train_step
optimizer.backward(loss)
File "/home/nmtuser/.local/lib/python3.8/site-packages/fairseq/optim/fairseq_optimizer.py", line 74, in backward
loss.backward()
File "/usr/lib/python3/dist-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/usr/lib/python3/dist-packages/torch/autograd/init.py", line 130, in backward
Variable._execution_engine.run_backward(

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [26, 112, 1536]], which is output 0 of AddBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
This could be incompatibility between fairseq 0.7.1 and Pytorch 1.6.0 or ModernMT uses fairseq in a way, which is no longer valid (see link). Now I have to decide, upgrade fairseq or resolve errors. What do you think?
I'm trying to fix the errors.

I have upgraded to fairseq 0.10.1 and tensorboardX 2.1. Also fixed some import module routine. After running the training, PC has gone loud!!! That is a good sign.
Additional steps required:

Remove torch==1.3.0 line from requirements.txt and change versions tensorboardX==2.1 fairseq==0.10.1
Edit ~/.local/lib/python3.8/site-packages/fairseq/utils.py change lines 441-444 from upper to lower:

            fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)
            if os.path.exists(fairseq_rel_path):
                module_path = fairseq_rel_path
            else:

            fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)
            module_parent, module_name = os.path.split(module_path)
            if os.path.exists(fairseq_rel_path):
                module_path = fairseq_rel_path
            elif not os.path.exists(module_parent):

Run the training ./mmt create en it examples/data/train/ --train-steps 10000

Hi all!
We have just released the support to CUDA 11 in the latest master commit!
We tested ModernMT on the new NVIDIA A100 with CUDA 11, PyTorch 1.7.1, and Fairseq 0.10.1.
In order to make ModernMT work with this configuration you will have to:

Install CUDA 11 (these are the only ones compatible with new NVIDIA chips)
install the new python dependencies: pip3 install -r requirements_cuda-11.txt (we recommend using a virtualenv)

ModernMT will automatically detect it is running with the newest Fairseq. However, this feature must be considered experimental for the moment because, unfortunately, we found out that:

Due to an issue with Fairseq 0.10.1 training won't work for the moment. We have already sent a pull request to Fairseq team and they have merged it! So it should be part of the next release of Fairseq.
Most importantly, we measured that the new Fairseq is significantly slower than its predecessor 0.7.1. So, at the end of the day, the A100 performed much worse in terms of speed compared to V100.

Ouch. So basically the best is to wait until the new architecture actually becomes faster?

I can confirm that ModernMT translation is working on RTX 3090 now.
I have encountered some errors while trying to install:

before running pip3 install -r requirements_cuda-11.txt run pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html otherwise error occurs
setup.py had to be modified on line 218 instead of requirements.txt place requirements_cuda-11.txt

It seems you don't need to install CUDA or CUDNN, because PyTorch have everything contained. I've only installed the GPU driver sudo apt install nvidia-driver-455.

I can confirm that ModernMT translation is working on RTX 3090 now.
I have encountered some errors while trying to install:

before running pip3 install -r requirements_cuda-11.txt run pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html otherwise error occurs
setup.py had to be modified on line 218 instead of requirements.txt place requirements_cuda-11.txt

It seems you don't need to install CUDA or CUDNN, because PyTorch have everything contained. I've only installed the GPU driver sudo apt install nvidia-driver-455.

How's speed compared to before?

How's speed compared to before?

I have run test translation with approx. 2000 character. The PC with 2x RTX 2080 Ti responded after 2.49s and the PC with 2x RTX 3090 responded after 2.71s. It seems there is about 10% performance dropdown.

I have run test translation with approx. 2000 character. The PC with 2x RTX 2080 Ti responded after 2.49s and the PC with 2x RTX 3090 responded after 2.71s. It seems there is about 10% performance dropdown.

Thank you very much for that insight! How about training?

Thank you very much for that insight! How about training?

I didn't tried yet, but according to

Due to an issue with Fairseq 0.10.1 training won't work for the moment.

it seems not working for now.

I didn't tried yet, but according to

Due to an issue with Fairseq 0.10.1 training won't work for the moment.

it seems not working for now.

@LoekvanKooten , just for your information, @davidecaroselli released a new MMT version (v4.9.5) a week ago with this new feature:
"Added support for CUDA 11 and Fairseq 0.10."
I thought I should let you know.

Yes, thank you, I noticed! However, at this moment it doesn't seem to give any added value due to speed issues. It does make ModernMT more future-proof though, which is a good thing.

* before running `pip3 install -r requirements_cuda-11.txt` run `pip3 install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html` otherwise error occurs

Doesn't work. This still gives:
Could not find a version that satisfies the requirement torch==1.7.1+cu110 (from -r requirements_cuda-11.txt (line 6)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0) No matching distribution found for torch==1.7.1+cu110 (from -r requirements_cuda-11.txt (line 6))

Hi Loek,
I believe pytorch has updated the package to torch==1.7.1+cu111, please try with that or, alternatively, you can force the installation of the old package with: pip3 install torch==1.7.1+cu110 -f.

@davidecaroselli
Solved! It is as @dagid4 said:

Edit ~/.local/lib/python3.8/site-packages/fairseq/utils.py change lines 441-444 from upper to lower:

            fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)             if os.path.exists(fairseq_rel_path):                 module_path = fairseq_rel_path             else:
            fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)             module_parent, module_name = os.path.split(module_path)             if os.path.exists(fairseq_rel_path):                 module_path = fairseq_rel_path             elif not os.path.exists(module_parent):
That solves the error. Make sure you use consistent spaces and tabs as indents, and use the correct indents.
This confirms that the newest version of ModernMT can train on a 3090, provided you use Ubuntu 20.04. This is on fairsec 0.10.1. 0.10.2 isn't even needed, as long as you edit utils.py as shown above.

What I can add already is that this training, slower as it may be (I didn't notice yet) seems to be much more stable. No crashes or anything. It just ploughs on and on. The constant crashes were a bit of a showstopper for me.

Okay, so thanks to @dagid4, the full procedure is as follows:

Install Ubuntu 20.04 LTS (this does not work on 18.04)
Install NVIDIA driver
sudo apt install nvidia-driver-455
Install java
sudo apt install -y openjdk-8-jdk
Install development tools
sudo apt install -y git maven cmake libboost-all-dev zlib1g-dev libbz2-dev
Clone ModernMT
git clone https://github.com/modernmt/modernmt.git modernmt && cd modernmt
Delete requirements.txt and rename requirements_cuda-11.txt to requirements.txt.
Run installation script
python3 setup.py
Compile binaries

pushd src
mvn clean install
popd
9. Edit ~/.local/lib/python3.8/site-packages/fairseq/utils.py change lines 441-444 from upper to lower:

        fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)
        if os.path.exists(fairseq_rel_path):
            module_path = fairseq_rel_path
        else:

        fairseq_rel_path = os.path.join(os.path.dirname(__file__), args.user_dir)
        module_parent, module_name = os.path.split(module_path)
        if os.path.exists(fairseq_rel_path):
            module_path = fairseq_rel_path
        elif not os.path.exists(module_parent):

Use a Python editor or make very sure you use indents, spaces and tabs consistently, else Python will throw an error.

Happy training!

My first engine has been trained and is actually working.

Thank you @LoekvanKooten !
I will add your guidelines to the INSTALL.md document!

Just to be sure, at the end of all this, is the 30XX series validated for both training and translating with ModernMT ?

Yes, and perfectly stable now. Not one crash. Not during translation. Not during training. Not once.


editing and configuring Webpack5 rules

break down the configuration process into distinct steps. Each step (or task) should focus on one aspect of the configuration, such as setting up source maps, handling specific file types, or optimizing the build process.

completed


Keystrokes are dropped on wormholed inputs

I have a repository where it is reproducible:
https://github.com/prashn64/focus-repro-wormhole
This README outlines steps to reproduce (Bug 3), but it's essentially when you have a text input inside a wormhole that you only activate/deactivate when the text content changes.  It seems that ripping the input out of the DOM and then re-placing it in the wormhole is causing a few keystrokes to fall through the cracks.
It's hard to see a case where it's too bad, but as the complexity of your component using the wormhole increases, the worse it gets.

may be related https://github.com/yapplabs/ember-wormhole/issues/40,


Wrong example at documentation page

Hi,
at documentation site https://github.com/feross/webtorrent/blob/master/docs/get-started.md we have a section called Download and save a torrent (in Node.js) where there's an example code.
This example works well but you're showing messages at console using console.error:
fs.writeFile(file.name, buffer, function (err) {
console.error('Error saving ' + file.name)
})
But it's downloading successfully.
Probably the best example should be
fs.writeFile(file.name, buffer, function (err) {
if (err) {
console.error('Error saving ' + file.name)
} else {
console.log('File saved ' + file.name);
}
})
As I could see at writeFile documentation (https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback) the 3rd parameters is for error, it's for a callback function.
This creates confusion with new developers using the library, because it's download successfully and at the console it show error messages.

thanks, fixed!


Update to Webpack 2 syntax

Changed "query" to "options" — reflecting the new, preferred syntax in
Webpack 2.

@jabacchetta Thanks a lot :)

@all-contributors please add @jabacchetta  for tool


VM1b camera error (v4l2.VIDIOC_S_CTRL) (Permission denied)

The following error started occurring at https://github.com/Labsmore/pyuscope/commit/a03a1a2be26d32a5b61c3a9931144d268e7a7ae7
$ ./app/argus.py --microscope lip-vm1b
pygame 2.5.1 (SDL 2.28.2, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
Camera 'Hayear Camera: HY-800B': selected /dev/video0
GRBL homing w/ hack, selected tool: 0
Sending home command 1
Homing successful after 41.9 sec. Ready to use!
Joystick: detected Logitech Extreme 3D pro
failed on Exposure Time, Absolute 157
<class 'PermissionError'>: [Errno 13] Permission denied
  File "/home/labsmore/pyuscope/uscope/gui/control_scroll.py", line 149, in gui_changed
    self.cs.raw_prop_write(self.config["prop_name"], val)
  File "/home/labsmore/pyuscope/uscope/gui/v4l_control_scroll.py", line 109, in raw_prop_write
    v4l2_util.ctrl_set(fd, name, val)
  File "/home/labsmore/pyuscope/uscope/v4l2_util.py", line 142, in ctrl_set
    fcntl.ioctl(fd, v4l2.VIDIOC_S_CTRL, control)

This probably won't get resolved without some access to your system that is showing an issue. However, I see you are using a joystick. Does this happen when you don't use a joystick?

Yes, happens without joystick as well.

I got a laptop here now that reproduces the issues. Observations:

Occurs in both wayland and cinnamon
Seems to happen to every other control? If I skip a control the next one is problematic
File descriptor seems constant / not corrupted
I tried allocating the query objects a little differently and didn't seem to make a difference
I tried disabling the non-base ranges and still got issues
Repeating the call on failure does not help

Next steps:

Hexdump the query structure?
More google searches
Can we make a reduced test case?

I've made a quick attempt to port the existing code to use a new API (https://github.com/antmicro/pyrav4l2). However they don't have any examples read / writing controls and I'm getting some unexpected output back from get_control_value() and I'm unclear if I'm using the API wrong or if this is a bug. I'll dig a bit deeper

fixed by switching to above framework


Data Structures Project

Blog Link

92/100
Great job! I love how you had images for all of the prompts and your code worked as expected. Next time, try being more organized and explain more on the algorithm functions better.


Change file mode before changing file owner

Change mode first when configuring remote control unix socket. Some
security systems might strip capability of changing other user's system
even to process with effective uid 0. That is done on Fedora by SELinux
policy and systemd for example. SELinux audit then shows errors, because
unbound tries modifying permissions of not own file. Fix just by mode
change as first step, make it owned by unbound:unbound user as the last
step only.
Related: rhbz#1905441

Relevant audit log would look like:
type=PROCTITLE msg=audit(12/11/2020 04:34:30.280:1043) : proctitle=/usr/sbin/unbound -d
type=PATH msg=audit(12/11/2020 04:34:30.280:1043) : item=0 name=/run/unbound/unbound.control.pipe inode=72069 dev=00:18 mode=socket,755 ouid=unbound ogid=unbound rdev=00:00 obj=system_u:object_r:named_var_run_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(12/11/2020 04:34:30.280:1043) : cwd=/etc/unbound
type=SYSCALL msg=audit(12/11/2020 04:34:30.280:1043) : arch=x86_64 syscall=chmod success=yes exit=0 a0=0x55c3d2ca00a0 a1=0660 a2=0x3e2 a3=0x0 items=1 ppid=1 pid=30404 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=unbound exe=/usr/sbin/unbound subj=system_u:system_r:named_t:s0 key=(null)
type=AVC msg=audit(12/11/2020 04:34:30.280:1043) : avc:  denied  { fsetid } for  pid=30404 comm=unbound capability=fsetid  scontext=system_u:system_r:named_t:s0 tcontext=system_u:system_r:named_t:s0 tclass=capability permissive=0
type=AVC msg=audit(12/11/2020 04:34:30.280:1043) : avc:  denied  { fsetid } for  pid=30404 comm=unbound capability=fsetid  scontext=system_u:system_r:named_t:s0 tcontext=system_u:system_r:named_t:s0 tclass=capability permissive=0

Taken from RH bug.

Merged the fix and also applied a similar fix to NSD, for its remote control unix socket code. That fix is in https://github.com/NLnetLabs/nsd/commit/877c8738a30c6aac05ed3ef0289924beeff7f9a1 . Thank you for the fix up!


Support markup in translated strings

Feature Description
In #1473 we are removing usage of createInterpolateElement from the @wordpress/element package for good, as we can't rely on it being available / provided to us by WordPress (see #1374). We can also not include the package in our final bundle due to licensing.
That means, if we want to use markup in translated / internationalized strings, we need to come up with our own solution.
Right now, due to this limitation, we cannot do things like this:
__('All data are treated in accordance with Google Privacy Policy.', 'web-stories')

or this:
__('Jump into the <Link href="...">editor</Link>.', 'web-stories')

Instead, we currently have to do workarounds like this:
{
  /* translators: First half of "Jump into the Editor" */
  _x('Jump into the', 'plugin activation', 'web-stories')
}
{' '}
<Link href={newStoryURL} onClick={onClick}>
  {__('Editor', 'web-stories')}
</Link>

But concatenating strings like this is bad for translators.
For reference, with createInterpolateElement it would look like this:
createInterpolateElement(
  __('Jump into the editor.', 'web-stories'),
  {
    a: <Link href={newStoryURL} onClick={onClick}>
  }
)

Additional Context

createInterpolateElement implementation in Gutenberg - uses a simple regex-based tokenizer
Original discussion about this in Gutenberg (recommended reading to better understand the problem)
react-interpolate-component - similar API than createInterpolateElement but less readable for translators
html-react-parser - like createInterpolateElement, but a bit larger in size (15.7 kB minified, 6.3 kB minified+gzipped)

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria
QA Instructions

Now that we have some actual translators working on the translation, this issue crops up a lot.
@o-fernandez, we should really prioritize this one, even though we don't actually know how to solve it yet.

Now that we have some actual translators working on the translation, this issue crops up a lot.

There are now also a lot more places in the code base where we'd need something like this.

Added some more details to the ticket description.
I feel like with html-react-parser it might be possible to fix this for 1.1.0

Ok, would be good to see if a pod can take it up in this sprint, let's see how it goes in our sprint kickoff today.

I have a small proof-of-concept ready for this.

Reopening because of these warnings:


Bluetooth audio device connected or disconnected on windows will crash.

axmol version: 2.1.0
devices test on: Windows 11/Windows 10
developing environments

NDK version: r23c
Xcode version: 14.2+
Visual Studio:

VS version: 2022 (17.9+)
MSVC version: 19.39+
Windows SDK version: 10.0.22621.0+

cmake version: 3.22
Steps to Reproduce:

Whenever a Bluetooth audio device is connected or disconnected, the program will crash.
it crash on source file hrtf.cpp.

Any idea ? Please help.

Maybe it's openal-soft issue?

I can't reproduce this issue

i can reproduce the issue every time.

make sure the audioengine is using, like playing bgm or sound.
while playing sound, then connected or disconnect the bluetooth audio device.

what device are you using?

i can reproduce the issue every time.

make sure the audioengine is using, like playing bgm or sound.
while playing sound, then connected or disconnect the bluetooth audio device.

I follow, the steps, but still can't reproduce

I will explain the steps in more detail. When the Bluetooth is disconnected or connected, it will not crash for a short period of time (but sometimes it will crash immediately), but after several operations in connected and disconnected, it will crash in different places. .

My device is an ordinary Bluetooth headset, and I originally thought it would only happen with Windows 11. Later, I tested it on Windows 10 and had the same problem.
It would be really weird if everyone couldn't reproduce it.

I test several  times, but chould'nt reproduce, my device is windows 11 23H2 , and Bluetooth device is Xiaomi Buds 3Pro

I tried the latest version 2.1.2, and the crash problem was indeed fixed, so I replaced the OpenAL Library with the version I am currently using, and the problem was solved too.
Thanks for everyone's discussion, the Bluetooth problem has been solved.

strange, the v2.1.0, v2.1.2 using same openal version: 1.23.1-e714c8f (8659)


Update cloudwatch, dynamodb, kinesis, s3, sns, ... to 2.17.71

Updates

software.amazon.awssdk:cloudwatch
software.amazon.awssdk:dynamodb
software.amazon.awssdk:kinesis
software.amazon.awssdk:s3
software.amazon.awssdk:sns
software.amazon.awssdk:sqs
software.amazon.awssdk:sts

from 2.16.104 to 2.17.71.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "software.amazon.awssdk" } ]

labels: library-update, early-semver-minor, semver-spec-minor

Superseded by #767.


Populate checker profiles

After the introduction of checker profiles described in #907 we should actually populate these with some checks.

Done.


searched list populates on profile page, css changes, handlebars styl…

…ing, attempted fix on displaying the user name but not functioning

This all looks great. The problem with the name showing up looks like it has to do with the handlebars variable in profile.handlebars. I think the “.name” needs to be removed from where it is in the pic I’m attaching.


helm: Update prometheus to v2.3.2

This one might depend on https://github.com/coreos/prometheus-operator/pull/1670 I'll rebase it when https://github.com/coreos/prometheus-operator/pull/1670 got merged.

There's no dependency on #1670 , but it's also already merged :slightly_smiling_face: .

Yep, had some other issue. CI is green now. :)
@gianrubio

@gianrubio done, added a note in the comment

Any updates on this?

Sorry for the inconvenience, but the prometheus-operator and kube-prometheus helm charts are currently being migrated to the upstream stable helm charts. Because of this we are currently not accepting any pull requests. We apologize for the inconvenience, but this is necessary for the migration to actually happen. We announced this here: https://github.com/coreos/prometheus-operator/tree/master/helm#warning
You can view the progress and help here and here.
Thank you for your understanding! :slightly_smiling_face:


unrecognized arguments

Hi there,
I am trying to run this program for the first time. However, I am running into an issue with the test data because it didn't seem to recognize the model file.
This is the error message I receive.
predict.py: error: unrecognized arguments: --model_file models/rdforestWithMHCModel.pickle
I unzipped the file and downloaded all dependencies - do you have any idea what the issue could be?
Best, Zita

Hi Zita,
I apologize for the mistake. I have updated the README.md file. Please change "--model_file" to "--modelfile" instead. Thank you so much for your help.
Best regards,
Mỹ Diễm.


10G interface space slowdown

on Gen2 the 10G interfaces speed seems to slow down randomly. not sure if its the switch thats its connected to or a problem with the NIC.
repo it by running a long run reboot test for a while

try the 4.9.52 -> 4.9.80 kernel bump. There was some ixgbe bugfixes that wen into 4.9.64. Suspicous about a 100Mb link advertisement. Maybe possible the link gets downgrated to 100Mb which fits with the slowdown speed level

seems 4.9.80 kernel made no difference, let it run overnight.
also it seems to slowly degrade. next suspect is the network bridge. possibly some kind of resource leak causing the slow and gradual slowdown

have narrowed it down to to unloading/loading the ixgbe driver and scping a small 1GB file. the problem manifests itself within a 1-2 minutes when the sequence is ran continusouly

same sequence on a different system, 100 runs LR and SR both no problem

tried the latest ixgbe driver -> same problem

disabled a bunch of ixgbe driver gear -> same problem

at this point the switch is highly suspcious. need to plug it into a different port on the 100G switch and/or directly to regression system interfaces to see any change in behaviour. As the repo case is simple should be clear where the problem is now

switch showing ~ 1% packet error rate on the rx port. smells like a phy phase issue..

urgh! transceiver / LC connector just needed cleaning seems resovled

problem re-occurs. bouncing the port on the switch makes no difference this time. replace the LC transciver and plug in the other split cable to the PPS port as back.

replace the 10G LR SFP seems better.


Edited regex for new github fine-grained personal access tokens

Hi everyone, GitHub has introduced fine-grained personal access tokens (PATs), now in beta preview, fixing a problem with tokens used in scripts to access GitHub repositories.
This pull request adds a match to the new token format (PAT) to the regular expression

@gwen001, hi. Can you approve my PR?

@gwen001 , update


No module named 'sklearn.neural_network.multilayer_perceptron'

Under the latest version of sklearn (0.24.2) it appears that the MLP modules shifted around a bit. As a result under the latest version, I got the following error when trying to run mCaller.
ModuleNotFoundError: No module named 'sklearn.neural_network.multilayer_perceptron'

Since you don't import this directly in the Python code, I think it comes from the pickle file like this.
My solution for this was to downgrade sklearn to 0.22, so maybe that should be specified as a requirement or otherwise a higher version should be specified with an updated pickle.

That's a pain. Thanks for letting me know. If you try one of the new models I added ("CAAY..." or "CRAA...") does the error still occur?

The new models don't seem to work with the older 0.22 sklearn. In the new version they give the following error
mCaller.py -m GATC -r e_coli.fa -d ~/anaconda3/envs/mcaller/lib/python3.6/site-packages/mCaller/CAAYNNNNNRTAC_model_6_m6A.pkl -e events.tsv -f fastq/pcr_ecoli.fastq.gz -b A

1 contigs
1 threads
Error: could not find sequence for reference contig contig
4a2c2f99-da0d-4d53-96f3-1573d778d0ae	782	ACCGGMTCGAT	0.8,-0.195,-1.1199999999999999,1.28,-1.99,-0.9225,18.939263322884013	- - Index or Key Error
dict_keys(['general']) dict_keys(['MG', 'MC', 'MA', 'MT', 'MM', 'MH', 'AT', 'AC', 'AG', 'AA', 'AM']) MT
'MH'
Traceback (most recent call last):
  File "/home/shians/anaconda3/envs/mcaller/lib/python3.6/site-packages/mCaller/extract_contexts.py", line 199, in extract_features
    mod_prob = model[twobase_model].predict_proba([diffs]) #TODO: call model only when batch ready to write
KeyError: 'MH'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shians/anaconda3/envs/mcaller/bin/mCaller.py", line 187, in <module>
    main()
  File "/home/shians/anaconda3/envs/mcaller/bin/mCaller.py", line 184, in main
    args.train,modelfile,args.skip_thresh,args.qual_thresh,args.classifier,training_tsv,args.plot_training)
  File "/home/shians/anaconda3/envs/mcaller/bin/mCaller.py", line 58, in distribute_threads
    extract_features(tsvname,refname,read2qual,nvariables,skip_thresh,qual_thresh,modelfile,classifier,0,endline=bytesize,train=train,pos_label=training_pos_dict,base=base,motif=motif,positions_list=positions_list)
  File "/home/shians/anaconda3/envs/mcaller/lib/python3.6/site-packages/mCaller/extract_contexts.py", line 222, in extract_features
    print(model[twobase_model].predict_proba([diffs]))
KeyError: 'MH'

I need to fix this, but for now, please change "twobase = True" to "twobase = False" in extract_contexts.py (line 134 in my version)

Hi,
I tried using mCaller tool and encountered given error:
ModuleNotFoundError: No module named 'sklearn.neural_network.multilayer_perceptron'.
Next, I tried given solution: change "twobase = True" to "twobase = False" in extract_contexts.py (line 130 in my version),
if type(model) != dict:
model = {'general':model} #for compatibility with previously trained model
twobase = False
else:
twobase = True
base_model = base_models(base,twobase))
but still the error could not be resolved.
Please help me in getting the issue resolved.
Thanks
Nihar

Hi Nihar,
Which version of scikit learn do you have installed? From the previous poster: "My solution for this was to downgrade sklearn to 0.22, so maybe that should be specified as a requirement or otherwise a higher version should be specified with an updated pickle." (in addition to the twobase = False change)

Hi,
Thanks for your response.
We have installed scikit learn version 0.24 and 0.22 as well. We also installed updated pickle version 5. But getting still same error.
Command used:
mCaller.py -p positions.txt -r Reference.fasta -d mCaller/r95_twobase_model_NN_6_m6A.pkl -e sample_eventalign.tsv -f sample.fastq -b A

Hi, it looks like the correct version is still not accessible. Are you using a package manager?
If you run python in a terminal then import sklearn as skl
print(skl.__version__) what does it show?

Hi,
Thanks for your suggestions. Finally I was able to generate some output using -m option. I have 2 queries now.

how to generate the positions,txt file which is one of the input file for 6ma methylation detection.
how to interpret these output columns generated using -m option by mCaller?
J02459.1	b49cc8bb-d98c-4f34-bfac-e7396b1f3dd0	3019	AGACGMTCTGG	0.07,2.64,1.78,6.32,12.53,7.1,7.424780316344464	+	m6A	0.94

Hi,
Can you please provide me with the output column descriptions and also let
me know how to generate positions.txt file for 6ma analysis.
Thanks
Nihar
On Wednesday, November 30, 2022, al-mcintyre @.***>
wrote:

Hi, it looks like the correct version is still not accessible. Are you
using a package manager?
If you run python in a terminal then import sklearn as skl
print(skl.version) what does it show?
—
Reply to this email directly, view it on GitHub
https://github.com/al-mcintyre/mCaller/issues/29#issuecomment-1331973876,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A3B6RF5BV7YSX76JST2XF5DWK4XTHANCNFSM47V5VSKA
.
You are receiving this because you commented.Message ID:
@.***>

Hi Nihar,
Please see the README file.
Positions file: "file with a list of positions at which to classify
bases (must be formatted as space- or tab-separated
file with chromosome, position, strand, and label if
training)" (this is optional, you can also try predicting based on motifs by using the -m option instead)
Output columns: "This returns a tabbed file with per-read predictions, where columns indicate chromosome, read name, genomic position, position k-mer context, features, strand, label, and probability of methylation predicted by mCaller for that position and read".

Hi,
Can you please answer my query regarding availability of next updated
version for mCaller tool?
Thanks
Nihar
On Friday, December 2, 2022, al-mcintyre @.***> wrote:

Hi Nihar,
Please see the README file.
Positions file: "file with a list of positions at which to classify
bases (must be formatted as space- or tab-separated
file with chromosome, position, strand, and label if
training)" (this is optional, you can also try predicting based on motifs
by using the -m option instead)
Output columns: "This returns a tabbed file with per-read predictions,
where columns indicate chromosome, read name, genomic position, position
k-mer context, features, strand, label, and probability of methylation
predicted by mCaller for that position and read".
—
Reply to this email directly, view it on GitHub
https://github.com/al-mcintyre/mCaller/issues/29#issuecomment-1334920346,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A3B6RF2WQG43SKKQRJQ3L6DWLGZZTANCNFSM47V5VSKA
.
You are receiving this because you commented.Message ID:
@.***>

Hi Nihar,
Unfortunately, there are no current plans for an updated version as everyone involved has moved on to other projects.

Hi,
I tried using mCaller tool and encountered some error:
ModuleNotFoundError: No module named 'sklearn.neural_network.multilayer_perceptron
Then I downgrade sklearn to 0.22 and change "twobase = True" to "twobase = False" in extract_contexts.py (line 130 in my version). But there are still problems:
`Traceback (most recent call last):
File "/public/home/yizhou/met/mcaller/mCaller-master/extract_contexts.py", line 199, in extract_features
mod_prob = model[twobase_model].predict_proba([diffs]) #TODO: call model only when batch ready to write
KeyError: 'general'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/public/home/yizhou/miniconda3/envs/mcaller/bin/mCaller.py", line 187, in
main()
File "/public/home/yizhou/miniconda3/envs/mcaller/bin/mCaller.py", line 183, in main
distribute_threads(args.positions,args.motif,args.tsv,read2qual,args.reference,num_refs,base,mod,args.threads,args.num_variables,
File "/public/home/yizhou/miniconda3/envs/mcaller/bin/mCaller.py", line 58, in distribute_threads
extract_features(tsvname,refname,read2qual,nvariables,skip_thresh,qual_thresh,modelfile,classifier,0,endline=bytesize,train=train,pos_label=training_pos_dict,base=base,motif=motif,positions_list=positions_list)
File "/public/home/yizhou/met/mcaller/mCaller-master/extract_contexts.py", line 222, in extract_features
print(model[twobase_model].predict_proba([diffs]))
KeyError: 'general'`
I don't know how to solve this error.
I hope someone kind can help me.
Thank you


Exception error when running on Server 2012 R2, works fine on Server 2008 R2

My code so far works on Server 2008 R2, but when I tried it on Server 2012 R2 I got completely different results.
PoshRSJob version (same on both servers): <IP_ADDRESS>
PowerShell Version Server 2008 R2
`PS> $PSVersionTable.PSVersion
Major  Minor  Build  Revision

5      0      10586  117`
PowerShell Version Server 2012 R2
`PS> $PSVersionTable.PSVersion
Major  Minor  Build  Revision

5      0      10586  117`
Error When Running on Server 2012 R2
`Exception calling "BeginInvoke" with "0" argument(s): "Cannot perform the operation because the runspace pool is not
in the 'Opened' state. The current state is 'Broken'."
At C:\Program Files\WindowsPowerShell\Modules\PoshRSJob\<IP_ADDRESS>\Public\Start-RSJob.ps1:406 char:17

            $Handle = $PowerShell.BeginInvoke()

            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : InvalidRunspacePoolStateException`

My Code
Ignore less than ideal module import, my module is a work in progress. :)
Note, $arrCustIDs comes from elsewhere, just an array of strings.
`ForEach ($Location in $StorageLocations) {
Write-Verbose "Collecting usage data for $($Location.StoragePath)."
Write-Verbose "Starting background jobs for each customer! Hold on to your butts!"
$NamePart = $Location.StorageClass
Write-Host "Collecting data for $NamePart." -ForegroundColor Green
$arrCustIDs|Start-RSJob -Name {"$($NamePart)$($)"} -ScriptBlock {
Param($StoragePath,$StorageClass,$ISO8601)
$StorageUsage = (Get-FolderSize -Path ($StoragePath + '' + $($)) -Precision 2).TotalGBytes
# Output of the job, drop it on the Pipeline.
New-Object PSObject -Property @{
CustomerID=$
StorageClass=$StorageClass
StorageUsage=$StorageUsage
ISO8601=$ISO8601
}
} -ModulesToImport "H:\My Documents\WindowsPowerShell\OP2_StorageBilling\OP2StorageBilling.psm1" -Throttle $MaxJobs -ArgumentList $Location.StoragePath,$Location.StorageClass,$ISO8601 | Wait-RSJob
}
Wait for jobs to finish before collecting results, then insert results into SQLite!
$result = Get-RSJob | Receive-RSJob
Get-RSJob | Remove-RSJob # Clean out the jobs, don't need them anymore! `

After a weekend and fresh eyes that haven't been staring at code for too many hours...

Set-ExecutionPolicy -ExecutionPolicy Unrestricted

Security warning
Run only scripts that you trust. While scripts from the internet can be useful, this script can potentially harm your computer. If you trust this script, use the Unblock-File cmdlet to allow the script to run without this warning message. Do you want to run .\Jobs-UpdateAll.ps1?
[D] Do not run  [R] Run once  [S] Suspend  [?] Help (default is "D"): r

Set-ExecutionPolicy -ExecutionPolicy Bypass
or use Unblock-File cmdlet as the Security warning says, no problems.
Sorry for the bug report that isn't a bug. Hopefully someone else will find my moment of "Duh" helpful. :+1:


vscode.dev Insiders colouring issue

The Insiders Edition of vscode.dev, does not show any colours on text inside the editor...
Example:

Version: 1.65.0-insider
Commit: 62de8dc3f78b8bef1eb62deeb093164a8d76c254
User Agent: Mozilla/5.0 (X11; CrOS x86_64 14324.80.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.102 Safari/537.36
Embedder: vscode.dev

I can repro this in VS Code Insiders on github.dev:

Yes, I see it is working now too. Thanks for the help!


Request for BuildTools_Full.exe times out, possibly b/c of proxy

When trying to npm i -g --production windows-build-tools, the postinstall script fails with a timeout when trying to download BuildTools_Full.exe (I'm running from an elevated cmd.exe on Windows 10):
C:\WINDOWS\system32>node --version
v6.4.0

C:\WINDOWS\system32>npm i -g --production windows-build-tools
(node:16288) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.

><EMAIL_ADDRESS>postinstall C:\Users\{{my user}}\AppData\Roaming\npm\node_modules\windows-build-tools
> node ./lib/index.js

Downloading BuildTools_Full.exe
Error: connect ETIMEDOUT <IP_ADDRESS>:80
{ Error: connect ETIMEDOUT <IP_ADDRESS>:80
    at Object.exports._errnoException (util.js:1026:11)
    at exports._exceptionWithHostPort (util.js:1049:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)
  code: 'ETIMEDOUT',
  errno: 'ETIMEDOUT',
  syscall: 'connect',
  address: '<IP_ADDRESS>',
  port: 80 }

I'm behind a corporate proxy (http://proxy:8080), have npm configured to use it, and npm works successfully with it.
Is windows-build-tools known to work behind proxies? Is there a way (e.g. via environment variables) to tell it the proxy to use? Am I missing something?

:tada: There's proxy support now, thanks for reporting! Use PowerShell to set a PROXY environment variable and let me know if it works - it should, but I don't have the means of actually testing it.
$env:PROXY = "Your Proxy"

You sir are a beast!
Works like a charm. Wish we'd ship fixes in 9 hours as well... great work!


New version: Optimisers v0.2.2

Registering package: Optimisers
Repository: https://github.com/FluxML/Optimisers.jl
Created by: @CarloLucibello
Version: v0.2.2
Commit: 385861188e81c7d7733fdf039d6f8884bae63b80
Reviewed by: @CarloLucibello
Reference: https://github.com/FluxML/Optimisers.jl/commit/385861188e81c7d7733fdf039d6f8884bae63b80#commitcomment-68068117
Description: Optimisers.jl defines many standard optimisers and utilities for learning loops.

@CarloLucibello did you skip the version on purpose?  What do you want to do?
[noblock]

ops sorry, it was a mistake, we can close this
[noblock]


use fairtally to generate an overview table of tests

see #264
this could make it easier to find the correct place where to add a new test

Just to clarify, this issue isnt about running fairtally to generate a results table, it is specifically meant to give howfairis developers some idea of which cases are covered by tests and which are not


Bug when adding Induction Smelter recipes

Consider following script:
'''
import mods.thermalexpansion.Smelter;
var certusQuartz = appliedenergistics2:item.ItemMultiMaterial;
var boneMeal = minecraft:dye:15;
var fusedQuartz  = EnderIO:blockFusedQuartz;
Smelter.addRecipe(1000, certusQuartz4 , boneMeal4 , fusedQuartz, null, 0);
Which should craft a Fused Quart from 4 Bonemeal and 4 certus quartz crystals.
Scrip loads fine, and NEI show the correct recipe.
However I can put the bone meal and the quartz only into the right input slot into the induction smelter :(
It will not accept any of it in the left input slot.
MineTweaker3 - MineTweaker 3 - 3.0.9B ( MineTweaker3-1.7.10-3.0.9C.jar )
modtweaker2 - Mod Tweaker 2 - 0.7 ( ModTweaker2-1.7.X-0.7.1.jar ) ( 0.7.3 will not work on my server)

What version of Thermal Expansion are you using?  The newest version has the ability to lock the left slot, so it may be locked without you knowing.

ThermalExpansion - Thermal Expansion - 1.7.10R4.0.0RC6

Ah ok that was it ... the "Unlock/Lock" Button is invisible in my ressourcepack!
Thanks for the hint!


ISPN-14815 Add an infinispan-dev-mode.xml configuration

https://issues.redhat.com/browse/ISPN-14815

@tristantarrant since 15 final is coming in very long time, a backport will be really helpful


Add Windows/DOS Batch file grammar

I reported an issue of this not being present with https://github.com/hackjutsu/Lepton
Response:

Does Powershell count?

The core team doesn't have a plan for ongoing development of new syntaxes (typically it's all we can do to keep up with support, issues, PRs, etc) - building support for new languages is all up to the larger community.  If you'd really like to see Idris, perhaps you know a developer who'd be willing to put in the time to develop and maintain a grammar for it?

Rather than leave these issues open to languish forever (as we have in the past) we're going to start closing them (to clean up the issues list).
I've moved this request to the wish list:
https://github.com/highlightjs/highlight.js/wiki/Grammar-Requests

For anyone who's considering contributing this, here's an MIT-licensed grammar description you can steal base yours off of:
https://github.com/microsoft/vscode-textmate/blob/975e2e3fad39172a3081d4004ccf56a4f9485297/test-cases/themes/syntaxes/Batch File.tmLanguage
It looks like that file is the grammar Sublime used, which was then harvested by Ace, which is what's used by Linguist, which is what Github uses… so it'll probably be at least a "serviceable" starting point


Fix some translations not properly falling back

Fixes #50650.

Thanks!

"Cancel" buttons in Project Manager dialogs are still untranslated. I guess this only fixes part of the problem.


control panel won't connect to community server over https

While trying to solve problem with fulltext search in CS (https://github.com/ONLYOFFICE/Docker-CommunityServer/issues/113),
on suggestion of @Carazyda I tried to install controlpanel. I have dificulty making it connect to existing installation of document & community server.  (everything runs in docker)
I suspect the problem might be that control panel is trying to connect to community server through http while it is running on https (thus won't return json).
root@controlpanel:/var/www/onlyoffice/controlpanel# cat /var/log/onlyoffice/web.controlpanel.12-31.log

2020-12-31 06:38:09 - error: http://onlyoffice.fritz.box/api/2.0/settings/controlpanel.json Unexpected token  in JSON at position 0
2020-12-31 06:38:09 - error: http://onlyoffice.fritz.box/api/2.0/portal.json Unexpected token  in JSON at position 0

(the correct address is https://.....) - when I try through web browser it returns html "access denied" on http:// and correct json data on https://.
Is there a way how to explain to control panel that community server is running on https? The run script obviously expects http :-(
root@controlpanel:/var/www/onlyoffice/controlpanel# cat run-controlpanel.sh

...
sed 's/http://onlyoffice-community-server/http://'${ONLYOFFICE_COMMUNITYSERVER_HOST}'/' -i www/config/production.json;
...

other issue you might want to know about is that there is error in doc on page
https://helpcenter.onlyoffice.com/installation/groups-install-docker.aspx:

...
sudo docker run --net onlyoffice -i -t -d --restart=always --name onlyoffice-control-panel
-v /var/run/docker.sock:/var/run/docker.sock
-v /app/onlyoffice/CommunityServer/data:/app/onlyoffice/CommunityServer/data
-v /app/onlyoffice/ControlPanel/data:/var/www/onlyoffice-controlpanel/Data
-v /app/onlyoffice/ControlPanel/logs:/var/log/onlyoffice-controlpanel
onlyoffice/controlpanel
...

^ the server expects mountpoints /var/www/onlyoffice/Data & /var/log/onlyoffice, not -controlpanel (I had to dig into the container to find the correct paths)

The control panel "starts" - listens on :80, but gives endless redirect to self responses (until web browser gives up with too many redirects)

Hello @loigu ControlPanel and CommunityServer should work each other on a local docker network, for example address http://onlyoffice-community-server, you can see at in config www/config/production.json. If communityserver don't available on this address - you can change it to correct one.

Hello @loigu ControlPanel and CommunityServer should work each other on a local docker network, for example address http://onlyoffice-community-server, you can see at in config www/config/production.json. If communityserver don't available on this address - you can change it to correct one.

So to further this discussion. I have been trying with no luck.  I have installed community server and control panel sharing the same network.  I get the same result in the log file.  Basically the resource /api/2.0/settings/controlpanel.json does not work.. calling it using curl from the control-panel docker instance returns
<!DOCTYPE html>

<meta charset="utf-8">
<title>Error</title>

Cannot GET /api/2.0/settings/controlpanel.json

This seems to apply to the following resources that the control-panel is trying to load from the community-server

<EMAIL_ADDRESS>api/2.0/portal.json
api/2.0/settings/companywhitelabel.json
api/2.0/settings/controlpanel.json

Beyond this, I have no idea why the community server requires the CONTROL_PANEL_PORT_80_TCP and CONTROL_PANEL_PORT_80_TCP_ADDR environment variables... shouldn't this be the other way around?


composite projects and ts-expect-error

Bug Report
composite TypeScript projects with allowJs: false and using @ts-expect-error to allow importing a .js file behaves unexpectedly.
The set up for this is kinda complex and I've put together a stripped down repository demonstrating it:
https://github.com/vitch/ts-composite-error
In this setup we have three packages:

package-a - imports a class from a .js file from package-b with @ts-expect-error and exports a subclass of it
package-b - provides that .js file for import
package-c - imports the subclass exported from package-a

When you try to build this app you get an error:

packages/package-a/declarations/src/index.d.ts:1:21 - error TS2307: Cannot find module 'ts-composite-error-b/foo' or its corresponding type declarations.
1 import { Foo } from 'ts-composite-error-b/foo';

Note that the error is present because the .d.ts from package-a is being type checked and it doesn't contain the @ts-expect-error comment that the source .ts file does.
🔎 Search Terms
@ts-expect-error, composite, .d.ts, comments
🕗 Version & Regression Information
I've reproduced this in 4.5.4, 4.7.4 and @next. I have no reason to believe that there was an older version where it wasn't a problem.

This is the behavior in every version I tried, and I reviewed the FAQ for entries about modules

⏯ Playground Link
Sorry - I couldn't figure out how to model the interaction between modules in a playground link so I made a repository:
https://github.com/vitch/ts-composite-error
💻 Code
I stripped it down as much as I could in:
https://github.com/vitch/ts-composite-error
🙁 Actual behavior
An error fired on the import
🙂 Expected behavior
The @ts-expect-error to be respected despite the additional "module hop"

This isn't a provided behavior of ts-expect-error. It can't prevent downstream consequences of the error you're suppressing.

Thanks for the quick response. It's the same with ts-ignore by the way but I guess that's also intentional...
Is there any workaround for this? It's something we're running into while incrementally converting a large monorepo to TypeScript...

What you'd want in this case is some commonly-referenced package declaring all the package paths you want to silence the error on
declare module "ts-composite-error-b/foo";

Thanks!


Simplify troubleshooting guide

Remove note about @sap modules installation (outdated/confusing, blog post from mid 2020).
Use cds-serve instead of cds run
Simpler wording + fixed typos

Broken link.  Please fix.

Fixed internally in cap/docs/commit/db3924aa9b3f92b6b2e8b952101f9bc3ba7975b8


Add NFTify to Showcase

Checklist

[x]  I have read the How to Contribute.
[x]  I have read the Showcase Requirements
[x]  I have run yarn build after adding my changes without getting any errors.

Showcase addition

Title: NFTify
Description: The leading no-code NFT marketplace solution aims to empower all Web3 businesses to launch and trade their NFT assets without friction, coding skills – at the lowest possible costs.
Website: https://nftify.network/
Tags:

marketplace
minting
nftproject

@thanhtran812 please first follow the recommendation on the repository front page: build the Portal to check your submission & then each time you make a structural change.
I haven't verified the appropriateness of all the tags but in a future commit I believe the wrong one was removed (see below).
Instead of a full screenshot, I would recommend using a detail of this web site for your cover image. Thumbnails will display, at the most, about 320 pixels of width so people won't be able to see much more than the background splash. How about changing to a zoom in on something both eye-catching and functional with text readable at this size?
I also don't know how many submissions are appropriate for this category & since @katomm did a category trim recently maybe he can give us some idea about that. 🧐

@rphair I have just changed the thumbnail into a 320px image and make some updated regarding your comments. Please check and let me know if there's any thing inappropriate. Thank you!

@thanhtran812 I appreciate your cooperation in standardising your listing.  I feel we wouldn't need another NFT creator but I think the number of white-label NFT creation services are fewer & therefore your service could be elegible.
It doesn't seem from the reading I've done that this service is connected to Cardano very much... I would invite other reviewers @rdlrt @katomm to see if they can verify that, and please also @thanhtran812 help me understand the Cardano connection.  Cardano isn't mentioned on the web site itself; only in a Document section: https://support.nftify.network/hc/en-us/sections/10339864961049-Cardano-network
... and the "white paper" (written & formatted like a sales pitch) also doesn't mention Cardano and says its Ethereum-based token is stakeable on a small number of chains which don't include Cardano.
So I can't approve this listing unless another reviewer can take a second look and see a primary Cardano connection that I may have missed.  My understanding at this point is that this is an Ethereum based service that has been made compatible with Cardano, which doesn't itself qualify this service for inclusion in the Developer Portal.

@thanhtran812 I appreciate your cooperation in standardising your listing. I feel we wouldn't need another NFT creator but I think the number of white-label NFT creation services are fewer & therefore your service could be eligible.
It doesn't seem from the reading I've done that this service is connected to Cardano very much... I would invite other reviewers @rdlrt @katomm to see if they can verify that, and please also @thanhtran812 help me understand the Cardano connection. Cardano isn't mentioned on the web site itself; only in a Document section: https://support.nftify.network/hc/en-us/sections/10339864961049-Cardano-network
... and the "white paper" (written & formatted like a sales pitch) also doesn't mention Cardano and says its Ethereum-based token is stakeable on a small number of chains which don't include Cardano.
So I can't approve this listing unless another reviewer can take a second look and see a primary Cardano connection that I may have missed. My understanding at this point is that this is an Ethereum based service that has been made compatible with Cardano, which doesn't itself qualify this service for inclusion in the Developer Portal.

Hello @rphair,
I would like to address your assumption that Cardano isn't featured in NFTify's marketing materials. We can provide you with website links, introduction videos as well as access to our portfolio. You can take a look to see Cardano is indeed highlighted as one of the main networks alongside Polygon, Binance, and Ethereum.

https://nftify.network/features/build-nft-store
https://nftify.network/services/concierge
https://www.youtube.com/watch?v=ZttnAm1h1T8
https://www.youtube.com/watch?v=11SuV9dnykg&t=5s (2:58)
Portfolio: https://docsend.com/view/txr95hhwqrkr8p9r (slide 13)

Btw, thank you so much for pointing out the absence of Cardano in our white paper. Actually, we no longer use this document to present to our customers or partners. We'll work on updating the white paper and publishing it on our website. I appreciate your time in reviewing our submission and look forward to hearing from you

@thanhtran812 yes, those additional materials might be helpful for other reviewers to establish enough of a relationship to Cardano for inclusion on the Developer Portal.  I still don't see a strong connection but I'll go along with other reviewers if they think the Cardano relationship is strong enough & will invite them to review your submission now.

Dear @rphair, @rdlrt, @katomm, @os11k, and @fill-the-fill,
I hope you guys had a chance to go through our submission and NFTify platform as well. we eagerly look forward to hearing your opinion and in case you need additional information from us to make the final decision, please don't hesitate to let me know.


Bump chartpress dependency to work around ruamel.yaml changes

Bring in a newer version of chartpress to fix build errors like https://travis-ci.org/jupyterhub/binderhub/jobs/416092817 due to ruamel.yaml changing the type of objects.
The changes this brings in https://github.com/jupyterhub/chartpress/compare/271c75e...2f26733f6ebf9e2b76bd9cfa3b8a8439daa0b16c

We broke the build https://travis-ci.org/jupyterhub/binderhub/jobs/416427639 after merging. Going to restart travis.

Ah this is the same bug that bit us in the littlest jupyterhub I think