possibly misaligned/aliased ivars?

(not sure this is the right place to ask, but here goes)
Hello,
I am trying to use ObjC with ARC, libdispatch and blocks on Linux.  I believe I am seeing a problem where ivars are referenced incorrectly, or are being aliased.  I have tried to reduce the problem to a very small example.
Last year, using Ubuntu 12.04 and clang 3.4 I built and used ObjC and libobjc2  successfully.  I am trying to recreate a similar working environment on Debian 8.  I installed clang from 'apt-get'.  It is version 3.5.
Here is the program that illustrates the problem.  The arrays are there to illustrate that gnustep-base seems to be working.  The ivars 'i' and 'j' are problematic.
// #import <Foundation/Foundation.h>

@interface Thing : NSObject
@property (readwrite) int i;
// @property int pad1;
// @property int pad2;
@property (readwrite) int j;
@property (readwrite) NSMutableArray * arr;
@end

@implementation Thing

- (id) init {
  if (self = [super init]) {
    _i = 5;
    _j = 17;
    _arr = [[NSMutableArray alloc] init];
    [_arr addObject:@"test1"];
    [_arr addObject:@"test2"];
  }
  return self;
}

@end

int main(int argc, char *argv[])
{

  NSLog(@"starting");

  NSMutableArray *thing = [[NSMutableArray alloc] init];
  [thing addObject:@"one"];
  [thing addObject:@"two"];
  [thing addObject:@"three"];
  NSLog(@"array:%@", thing);
  thing = [[NSMutableArray alloc] init];
  NSLog(@"array:%@", thing);

  Thing *t = [[Thing alloc] init];
  NSLog(@"Thing:%@", t);
  NSLog(@"i:%u", t.i);
  NSLog(@"j:%u", t.j);
  NSLog(@"arr:%@", t.arr);
}

On my Ubuntu 14.04 install it prints this:

2016-11-10 08:40:43.315 errorexample2[13183:13183] starting
2016-11-10 08:40:43.330 errorexample2[13183:13183] array:(one, two, three)
2016-11-10 08:40:43.332 errorexample2[13183:13183] array:()
2016-11-10 08:40:43.332 errorexample2[13183:13183] Thing:<Thing: 0x193d1c0>
2016-11-10 08:40:43.332 errorexample2[13183:13183] i:5
2016-11-10 08:40:43.332 errorexample2[13183:13183] j:17
2016-11-10 08:40:43.332 errorexample2[13183:13183] arr:(test1, test2)

On my Debian 8 install it prints this, the i and the j are aliased somehow, and they share the same value.

2016-11-10 08:38:46.121 errorexample2[26516:26516] starting
2016-11-10 08:38:46.122 errorexample2[26516:26516] array:(one, two, three)
2016-11-10 08:38:46.122 errorexample2[26516:26516] array:()
2016-11-10 08:38:46.122 errorexample2[26516:26516] Thing:<Thing: 0x1f1e070>
2016-11-10 08:38:46.123 errorexample2[26516:26516] i:17
2016-11-10 08:38:46.123 errorexample2[26516:26516] j:17
2016-11-10 08:38:46.123 errorexample2[26516:26516] arr:(test1, test2)

If I put in the padding properties, then things get even stranger.

2016-11-10 08:46:46.126 errorexample2[26553:26553] starting
2016-11-10 08:46:46.127 errorexample2[26553:26553] array:(one, two, three)
2016-11-10 08:46:46.127 errorexample2[26553:26553] array:()
2016-11-10 08:46:46.127 errorexample2[26553:26553] Thing:<Thing: 0x9d5070>
2016-11-10 08:46:46.128 errorexample2[26553:26553] i:10303216
2016-11-10 08:46:46.128 errorexample2[26553:26553] j:10303216
2016-11-10 08:46:46.128 errorexample2[26553:26553] arr:(test1, test2)

The recipe I've followed for building libobjc2 and libdispatch are as follows.  I've repeated this a few times.

sudo apt -y install clang git ninja cmake libffi-dev libxml2-dev libgnutls28-dev libicu-dev libblocksruntime-dev libkqueue-dev libpthread-workqueue-dev autoconf libtool libjpeg-dev libtiff-dev libffi-dev libcairo-dev libx11-dev libxt-dev libxft-dev libc++-dev

export CC=clang
export CXX=clang

# Checkout sources
git clone https://github.com/nickhutchinson/libdispatch.git
git clone https://github.com/gnustep/libobjc2.git
git clone https://github.com/gnustep/make
git clone https://github.com/gnustep/base.git
git clone https://github.com/gnustep/gui.git
git clone https://github.com/gnustep/back.git

cd libdispatch
mkdir build; cd build
make
sudo -E make install

cd libobjc2
cmake ../ -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang -DCMAKE_ASM_COMPILER=clang -DTESTS=OFF
cmake --build .
sudo -E make install

export LDFLAGS=-L/usr/local/lib

cd make
./configure --enable-debug-by-default --with-layout=gnustep --enable-objc-nonfragile-abi
make
sudo -E make install
sudo ldconfig

cd base
make clean
./configure

export LD_LIBRARY_PATH=/usr/local/lib:/usr/GNUstep/Local/Library/Libraries

Now build the program

clang -g -o errorexample2 -x objective-c -fobjc-runtime=gnustep-1.8.1 -v -I/usr/local/include/objc -I/usr/GNUstep/Local/Library/Headers -f
objc-arc errorexample2.m Journal.m -L/usr/local/lib -L/usr/GNUstep/Local/Library/Libraries  -lobjc -lgnustep-base -ldispatch
Any ideas or pointers are appreciated.
Thanks

This was a bug introduced recently.  See: 25c7038e.
If you're not using SSE or similar vectors, then 6b285936 should work correctly.

David -
thanks so much for your quick response.  That was indeed the problem.
I began the investigation with a larger program that was exhibiting very mysterious behavior.  Using the commit you showed, it seems to be working correctly now.
Much Thanks!


[Network] Added missing paths to LoadBalancer, NetworkInterface and VirtualNetworkGateway

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/1468

Codecov Report

Merging #1338 into master will decrease coverage by 0.17%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1338      +/-   ##
==========================================
- Coverage   56.05%   55.87%   -0.18%
==========================================
  Files        2692     2708      +16
  Lines       71250    71810     +560
==========================================
+ Hits        39936    40126     +190
- Misses      31314    31684     +370

Impacted Files
Coverage Δ

...7_06_01/models/effective_network_security_group.py
50% <0%> (-7.15%)
:arrow_down:

...17_06_01/models/effective_network_security_rule.py
21.05% <0%> (-5.62%)
:arrow_down:

.../operations/virtual_network_gateways_operations.py
17.98% <0%> (-1.09%)
:arrow_down:

...twork/v2017_06_01/operations/subnets_operations.py
97.05% <0%> (ø)
:arrow_up:

...re/mgmt/network/v2017_06_01/models/route_filter.py
50% <0%> (ø)
:arrow_up:

...work/v2017_06_01/models/virtual_network_gateway.py
100% <0%> (ø)
:arrow_up:

...ork/v2017_06_01/models/vpn_client_configuration.py
100% <0%> (ø)
:arrow_up:

...2017_06_01/operations/security_rules_operations.py
98% <0%> (ø)
:arrow_up:

...06_01/models/virtual_network_gateway_connection.py
20.83% <0%> (ø)
:arrow_up:

.../virtual_network_gateway_connections_operations.py
10.91% <0%> (ø)
:arrow_up:

... and 50 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 61c3829...b8892db. Read the comment docs.

Replaced by https://github.com/Azure/azure-sdk-for-python/pull/1373


Signed Firefox release

Hello,
Thanks for maintaining an updated fork of this extension. Any chance you could release a signed version for Firefox?

Thanks.

Definitely not a chance. I know it's quite annoying the way Firefox is managing temporary extension, but the only feasible way is to ask @alexadam to merge or whatever this fork to the main branch...

Well, it was worth a try. Thanks for explaining.


As a developer, I need to add a "loading results" message to the graphics

-Where:
Search tool, timeline chart
Inspector tool, timeline chart
-Problem:
Sometimes the graphics take long to load the data, creating confusion when the values don't change immediately. Example: Search tool, KB collection, switching from absolute to relative view for the whole collection takes a long time (the first time), and there is no indication that the application is working on it. See screencast.

@lilimelgar @avlisodraude cannot reproduce this one, so closing it for now


[$250] mWeb - Chat - In thread page, after sending a message edited text disappears.

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: V9. 0.75-0
Reproducible in staging?: Yes
Reproducible in production?: Yes
Issue reported by: Applause Internal Team
Device used: Redmi note 10s android 13
App Component: Chat Report View
Action Performed:

Go to https://staging.new.expensify.com/home
Open a chat
Send a message
Edit the message
Create a reply in thread
In thread page, send a message
Tap header sub link
Tap reply link

Expected Result:
In thread page, after sending a message edited text must not disappear.
Actual Result:
In thread page, after sending a message edited text disappears in parent message. On revisiting using reply link, edited text again displayed in oatent message.
Workaround:
Unknown
Platforms:

[ ] Android: Standalone
[ ] Android: HybridApp
[x] Android: mWeb Chrome
[ ] iOS: Standalone
[ ] iOS: HybridApp
[ ] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/user-attachments/assets/3f6cb8a6-2ade-4f3c-859a-7faa8140ade3
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~021867654489912774933
Upwork Job ID:<PHONE_NUMBER>912774933
Last Price Increase: 2024-12-13

Issue OwnerCurrent Issue Owner: @Ollyws

BE issue https://github.com/Expensify/App/issues/52876

Proposal
Please re-state the problem that we are trying to solve in this issue.
Chat - In thread page, after sending a message edited text disappears
What is the root cause of that problem?
When adding a comment, the AddComment API is called. However, the response from this API is missing the isEdited field inside reportActions_<report_parent_action_id>.
What changes do you think we should make in order to solve the problem?
In the TextCommentFragment we can pass the originalMessage to this component and update the condition for the edited text to: (!!fragment?.isEdited || originalMessage?.edits?.length > 0)
const originalMessage = ReportActionsUtils.getOriginalMessage(action)
https://github.com/Expensify/App/blob/0d17435fa6e7fa79865de14aaec08e4b1240386c/src/pages/home/report/comment/TextCommentFragment.tsx#L127
What specific scenarios should we cover in automated tests to prevent reintroducing this issue in the future?
N/A
What alternative solutions did you explore? (Optional)
Fix via BE

BE issue #52876

However, I have a way to fix it via FE

Sweet, I'll mark it internal


Getting a very vague "VCF Parser: could not open file"

Hey!
I keep getting this vague message on merge -- "VCF Parser: could not open file: (file path)"... for files that definitely exist and definitely are properly formatted VCFs. I cannot figure out why, but it will not take any of my files. Tried this in both cygwin (using both cygwin style and windows style paths) and Fedora - didn't work. I'd been using SURVIVOR for a couple years on the cluster at work and never had this issue...
Is this a common issue for people trying to compile/run this from home? Or is there some quirk to the parser that requires I modify the files in some way?

Hi,
thanks for reaching out.

Is there an empty line maybe in the files file?
Can you confirm that an ls on the file path as it is in the files file works?
Do you have read permissions on these files?
Can you past the full output?

Thanks
Fritz

Sure!
I didn't see any empty lines in the files. Also tried adding a newline at the end because some parsers insist upon that, and that didn't work either.
Here's the output + an ls:
[bhospelhorn@localhost svproject]$ DataDir="/home/bhospelhorn/Documents/data/svproject/GIAB" [bhospelhorn@localhost svproject]$ SurvivorExe="/home/bhospelhorn/Documents/Git/SURVIVOR/Debug/SURVIVOR" [bhospelhorn@localhost svproject]$  [bhospelhorn@localhost svproject]$ MergeInList=${DataDir}"/NA12878/Analysis/merge2020/paths.txt" [bhospelhorn@localhost svproject]$ MergeOutStem=${DataDir}/"NA12878/Analysis/merge2020/NA12878_" [bhospelhorn@localhost svproject]$  [bhospelhorn@localhost svproject]$ MaxDistBool=0 [bhospelhorn@localhost svproject]$ ConsiderType=0 [bhospelhorn@localhost svproject]$ ConsiderStrand=1 [bhospelhorn@localhost svproject]$ DistEstOnSize=1 [bhospelhorn@localhost svproject]$ ${SurvivorExe} merge ${MergeInList} ${MaxDistBool} 2 ${ConsiderType} ${ConsiderStrand} ${DistEstOnSize} 50 ${MergeOutStem}"Intersect_min2.vcf" VCF Parser: could not open file: /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_mtsinai.vcf [bhospelhorn@localhost svproject]$ ls -l /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_mtsinai.vcf -rw-rw----. 1 bhospelhorn bhospelhorn 4988638 Mar 20 12:30 /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_mtsinai.vcf
And paths.txt consists of
/home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_mtsinai.vcf /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_mills_liftover.vcf /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_svclassify_del.vcf /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_svclassify_ins.vcf /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_10x.vcf /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/NA12878_dbvar.vcf
Just playing with merging some of the GIAB callsets together. Some of the VCFs I generated, some I used as-is... none of them appear to work - it just rejects the first file in paths.txt and exits.

Can you confirm that there is no space or so after the name in the file?
I usually use an
ls /home/bhospelhorn/Documents/data/svproject/GIAB/NA12878/Analysis/merge2020/*vcf > files
This may sound stupid but many people and I are using this almost daily so something seems to be going on since otherwise everyone would complain..
Thanks
Fritz

Yup, there it is.
 Thought repasting it line by line in the Linux text editor would take care of that but apparently not haha... Wouldn't THINK I would need to constantly be checking my text files in a hex editor.... gr...
Ugh, thank you so much for your help still. Since the message said there was a parser error (sometimes "can't open" is also used for "can't read"), I just really didn't know if it was a path formatting issue or if there really was something odd about these files or running it on Fedora or something.
I'm so sorry for bugging you with this - I got really stumped since I couldn't figure out what was wrong.

No worries happens to us all. Glad it was something easy :) .
Cheers
Fritz


[API] add endpoint for single comment

like github has "latest_comment_url": "https://api.github.com/repos/octokit/octokit.rb/issues/comments/123",

useful for #4811


CB-8326 livy 3 added as a knox service and also added as supported se…

…rvice. Now we will be able to attach the right parcel to the cluster in case of livy 3
See detailed description in the commit message.

+1 LGTM (knox changes: https://gerrit.sjc.cloudera.com/c/cdh/knox/+/93119)


PDF attachments not opened on iOS

Attach (recommended) or Link to PDF file here:
any pdf that has pdf attachments
Configuration:

Web browser and its version: Safari latest version
Operating system and its version: iOS 12.2
PDF.js version: v<IP_ADDRESS>.3.
Is a browser extension: no

Steps to reproduce the problem:

Open pdf with attachments
Open a pdf attachment

What is the expected behavior? (add screenshot)
What went wrong? (add screenshot)

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):
The pdf viewer does not open pdf attachments on ios, any browser you use. On android it opens them without problems.

The pdf viewer does not open pdf attachments on ios, any browser you use. On android it opens them without problems.

That would suggest a platform issue, rather than a problem with the PDF.js default viewer; hence this issue may be INVALID from the PDF.js point of view.
Furthermore, note that the default viewer was developed mainly for desktop usage and may not be a good fit for mobile devices.


'Build completes successfully or spit out 500' util

Mostly of use for wrapping views that depend on the build working

This is heavily related to #45

Too messy, too much coupling to particular server framework's API.


Test Failure on OSX 10.15.7 x86_64

OSX 10.15.7 x86_64
rustc 1.51.0-nightly (8a6518427 2021-01-16)
cargo 1.51.0-nightly (a73e5b7d5 2021-01-12)
Terminal 2.10
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin19)
commit 0bb5adad3f5bfe55814389177b2e41
cargo test --all
...
  Undefined symbols for architecture x86_64:
            "___stdbuf_get_stderr", referenced from:
                ___stdbuf in libstdbuf.pdfpzpkbkd0wkn.rcgu.o
            "___stdbuf_get_stdin", referenced from:
                ___stdbuf in libstdbuf.pdfpzpkbkd0wkn.rcgu.o
            "___stdbuf_get_stdout", referenced from:
                ___stdbuf in libstdbuf.pdfpzpkbkd0wkn.rcgu.o
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error

error: could not compile `uu_stdbuf_libstdbuf`

cargo test
# successful but terminal is hosed - have to reset

Reran on zsh 5.7.1 (x86_64-apple-darwin19.0).
cargo test
# success this time, does not hose shell

cargo test --all
# same linker error

Reran on zsh 5.7.1 (x86_64-apple-darwin19.0).
cargo test
# success this time, does not hose shell

cargo test --all
# same linker error


Dispatch closure inside console command not working

Laravel Version: 5.7.8
PHP Version: 7.1.3

Description:
Dispatching a callback from console command class has an issue related to Carbon.
Error output:
  Exception trace:

  1   Carbon\Carbon::__set("date", "2019-09-21 20:26:19.267166")
      [internal]:0

  2   ReflectionProperty::setValue(Object(Illuminate\Support\Carbon), "2019-09-21 20:26:19.267166")
      /Users/apple/Code/nairaex-v2/vendor/opis/closure/src/SerializableClosure.php:431

Steps To Reproduce:

create new artisan command: php artisan make::command FooCommand
create FooCommand::TestAnyThing method
dispatch a callback:

dispatch(function(){
  $this->TestAnyThing();
});

Hey there,
Unfortunately we don't support this version anymore. Please check out our support policy on which versions we are currently supporting. Can you please try to upgrade to the latest version and see if your problem persists? We'll help you out and re-open this issue if so.
Thanks!

FWIW I was having the same issue but it turned out I had switched from
the database driver to sync without running php artisan config:cache, once I did that dispatch worked as you'd expect from a console command.


Error Start

Please provide a brief summary of the bug
A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f197d78d6f4, pid=1, tid=34

JRE version: OpenJDK Runtime Environment Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1) (build <IP_ADDRESS>+1)
Java VM: OpenJDK 64-Bit Server VM Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
Problematic frame:
V  [libjvm.so+0x7396f4]  G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x14d4

Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/container/core.1)

An error report file with more information is saved as:
/home/container/hs_err_pid1.log
[thread 51 also had an error]
[thread 53 also had an error]
[thread 52 also had an error]

If you would like to submit a bug report, please visit:
https://github.com/adoptium/adoptium-support/issues

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f197e3ca898]
Please provide steps to reproduce where possible
A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f197d78d6f4, pid=1, tid=34

JRE version: OpenJDK Runtime Environment Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1) (build <IP_ADDRESS>+1)
Java VM: OpenJDK 64-Bit Server VM Temurin-<IP_ADDRESS>+1 (<IP_ADDRESS>+1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
Problematic frame:
V  [libjvm.so+0x7396f4]  G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x14d4

Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/container/core.1)

An error report file with more information is saved as:
/home/container/hs_err_pid1.log
[thread 51 also had an error]
[thread 53 also had an error]
[thread 52 also had an error]

If you would like to submit a bug report, please visit:
https://github.com/adoptium/adoptium-support/issues

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f197e3ca898]
Expected Results
press start snow error and stop automat
Actual Results
no starting
What Java Version are you using?
java --19
What is your operating system and platform?
ununtu 20
How did you install Java?
no
Did it work before?
no

Did you test with the latest update version?
no

Did you test with other Java versions?
java --19

Relevant log output
No response

Could be https://bugs.openjdk.org/browse/JDK-8274053 which is not backported to 17.

Today, we ran into the same Problem
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff936b30214, pid=7, tid=51
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.10+7 (17.0.10+7) (build 17.0.10+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (17.0.10+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x73b214]  G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x14d4
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /usr/local/tomcat/core.7)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=
ALL-UNNAMED -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.web
resources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 --add-opens=java.base/sun.util.locale=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED -Dorg.apache.catalina.STRICT_SERVLET_COMPLIANCE=false -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Djava.security.egd=file:///dev/urandom -XX:ErrorFile=/var/log/java/hs_err_wtf.log -Xms7500M -Xmx10000M -Xss384k -XX:+UseG1GC -XX:G1HeapRegionSize=8M -XX:MaxGCPauseMillis=400 -XX:Initiat
ingHeapOccupancyPercent=1 -XX:ConcGCThreads=12 -XX:ParallelGCThreads=24 -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/java/heapdump-java/ -XX:+UseStringDeduplication -Xlog
:gc*,gc+stringdedup*=trace,gc+ref*=trace,gc+heap*=trace,gc+phases*=trace,gc+region*=debug,gc+marking*=debug,gc+remset*=debug,gc+task+stats*=trace,gc+task*=debug,gc+ergo*=trace,gc+age*=trace:file=/var/log/java/java-gc.log:tags,u
ptime,time,level:filecount=5,filesize=250m -Xlog:safepoint*:file=/var/log/java/java-safepoints.log:tags,uptime,time,level:filecount=5,filesize=50m -Denv-name=**** -Dspring.profiles.active=springMobile,elasticsearch,es-imag
e-service,production -Dlog4j2.isThreadContextMapInheritable=true -Dignore.endorsed.dirs= -Dcatalina.ba
se=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

Host: AMD EPYC 7502P 32-Core Processor, 64 cores, 503G, Ubuntu 22.04.3 LTS
Time: Thu Feb  1 19:55:25 2024 CET elapsed time:<PHONE_NUMBER>81 seconds (1d 12h 7m 6s)

---------------  T H R E A D  ---------------

Current thread (0x00007ff89800e650):  GCTaskThread "GC Thread#8" [stack: 0x00007ff8f03dd000,0x00007ff8f04dd000] [id=51]

Stack: [0x00007ff8f03dd000,0x00007ff8f04dd000],  sp=0x00007ff8f04db9b0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x73b214]  G1ParScanThreadState::trim_queue_to_threshold(unsigned int)+0x14d4
V  [libjvm.so+0x756e20]  G1ScanHRForRegionClosure::scan_heap_roots(HeapRegion*)+0x530
V  [libjvm.so+0x74e6d7]  G1RemSet::scan_heap_roots(G1ParScanThreadState*, unsigned int, G1GCPhaseTimes::GCParPhases, G1GCPhaseTimes::GCParPhases, bool)+0x1a7
V  [libjvm.so+0x6f6d4b]  G1EvacuateRegionsTask::scan_roots(G1ParScanThreadState*, unsigned int)+0x4b
V  [libjvm.so+0x6f75a3]  G1EvacuateRegionsBaseTask::work(unsigned int)+0x83
V  [libjvm.so+0xf1431f]  GangWorker::loop()+0x5f
V  [libjvm.so+0xf1437f]
V  [libjvm.so+0xe633b8]  Thread::call_run()+0xa8
V  [libjvm.so+0xc24791]  thread_native_entry(Thread*)+0xe1

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000125cf0790

Registers:
RAX=0x00007ff930056e44, RBX=0x00007ff82405faa0, RCX=0x00007ff9300572da, RDX=0x0000000000000000
RSP=0x00007ff8f04db9b0, RBP=0x00007ff8f04dba40, RSI=0x0000000000000003, RDI=0x000000000000024b
R8 =0x0000000000000001, R9 =0x00007ff8a3112500, R10=0x0000000000000000, R11=0x0000000000000017
R12=0x000000076726aafd, R13=0x0000000125cf0790, R14=0x00007ff9376f832d, R15=0x00007ff9376b1330
RIP=0x00007ff936b30214, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Register to memory mapping:

RAX=0x00007ff930056e44 points into unknown readable memory: 00 00 00 00
RBX=0x00007ff82405faa0 points into unknown readable memory: 0x00007ff937600ef0 | f0 0e 60 37 f9 7f 00 00
RCX=0x00007ff9300572da points into unknown readable memory: 00 00 00 00 00 00
RDX=0x0 is NULL
RSP=0x00007ff8f04db9b0 points into unknown readable memory: 0x00000000004485ab | ab 85 44 00 00 00 00 00
RBP=0x00007ff8f04dba40 points into unknown readable memory: 0x00007ff8f04dbb90 | 90 bb 4d f0 f8 7f 00 00
RSI=0x0000000000000003 is an unknown value
RDI=0x000000000000024b is an unknown value
R8 =0x0000000000000001 is an unknown value
R9 =0x00007ff8a3112500 is pointing into metadata
R10=0x0 is NULL
R11=0x0000000000000017 is an unknown value
R12=0x000000076726aafd is pointing into object: java.util.concurrent.ConcurrentHashMap$Node
{0x000000076726aae8} - klass: 'java/util/concurrent/ConcurrentHashMap$Node'
 - ---- fields (total size 4 words):
 - final 'hash' 'I' @12<PHONE_NUMBER> (7f9b5ebf)
 - final 'key' 'Ljava/lang/Object;' @16  "co.elastic.apm.exception"{0x00000005db13f200} (bb627e40)
 - volatile 'val' 'Ljava/lang/Object;' @20
[error occurred during error reporting (printing register info), id 0xb, SIGSEGV (0xb) at pc=0x00007ff936c20e9a]

Application runs in an dockerized tomcat.  Since few weeks we struggling also with other issues like
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2d95bf3498, pid=7, tid=51
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.10+7 (17.0.10+7) (build 17.0.10+7)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (17.0.10+7, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x760498]  G1SATBMarkQueueSet::filter(SATBMarkQueue&)+0x78
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /usr/local/tomcat/core.7)
#
# If you would like to submit a bug report, please visit:
#   https://github.com/adoptium/adoptium-support/issues
#

We don't know if it's just a coincidence that different problems occur during GC.
Maybe someone sees an unfavorable GC option that should be changed for JDK17.
We have been using JDK17 (tomcat:9-jdk17-openjdk) since July 2023 and so far there have been no problems. Due to a recommendation regarding another problem, we switched to the latest Temurin (tomcat:9-jdk17) on January 30, 2024. Since then there have been various problems related to GC.

@thaarbach Any luck with the very latest 17.0.11 based images?

@karianna sorry, didn't see your comment. The error disapear with newer image. The latest which we use is 17.0.12.


[discussion] On Upsert Hybrid Tables

I recently learnt that Pinot allows creating Hybrid tables where the Realtime part is a Upsert table, with the caveat that the primary key dedup is not done across the Realtime and Offline part.
Just wanted to make sure that this is known across the community so we don't end up removing support for this in the future.

imo, the solution for different customer depends on the requirements and ability of each customer.

currently open source doesn’t constraint the usage of hybrid table for upsert. you can setup hybrid table with realtime upsert table and compacted offline table with the same name.
In this use case the offline part we do not upserting based on Pinot. It’s very expensive to host the offline part PK indexes into the memory. The customer is doing upserting in the spark job.
Since the algorithm is different in the realtime part and offline part, they customer is willing to merge one records from realtime part and one record from offline part for their final result.

We are happy to gather feedback and concerns from open source.

is there any documentation of this feature and how would it behave during realtime-offline merge? for example if the realtime part is configured to be partial upsert how would that work?

This is not officially supported because of the known caveats, and it is quite hard to manage. User will need to create the post-upsert segments for offline table on their own.
We will likely stop supporting this model because the correct approach should be to use real-time only table, and directly push segments into it. Do you see issues when using real-time only table for your use case?


python311Packages.aioambient: 2023.12.0 -> 2024.01.0

Diff: https://github.com/bachya/aioambient/compare/refs/tags/2023.12.0...2024.01.0
Changelog: https://github.com/bachya/aioambient/releases/tag/2024.01.0
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 280228 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.ambient_station
python310Packages.aioambient
python310Packages.aioambient.dist
python311Packages.aioambient
python311Packages.aioambient.dist

Result of nixpkgs-review pr 280228 run on aarch64-linux 1

4 packages built:

python310Packages.aioambient
python310Packages.aioambient.dist
python311Packages.aioambient
python311Packages.aioambient.dist

Result of nixpkgs-review pr 280228 run on aarch64-linux 1

4 packages built:

python310Packages.aioambient
python310Packages.aioambient.dist
python311Packages.aioambient
python311Packages.aioambient.dist


Quick Fix import puts local imports with third-party imports

Environment data

Language Server version: 2023.9.20
OS and version: darwin arm64
Python version (and distribution if applicable, e.g. Anaconda): 3.11.5
python.analysis.indexing: true
python.analysis.typeCheckingMode: strict

Using poetry 1.6.1, python and packages installed in venv
Code Snippet

Expected behavior
Quick Fix-injected local imports are grouped together after third party imports
Actual behavior
Quick Fix-injected local imports are mixed in with third party imports
Logs
2023-09-23 13:59:37.877 [info] (87036) [BG(1)] analyzing: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py ...
2023-09-23 13:59:37.877 [info] (87036) [BG(1)]   parsing: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (2ms)
2023-09-23 13:59:37.877 [info] (87036) [BG(1)]   binding: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (1ms)
2023-09-23 13:59:37.897 [info] (87036) [BG(1)]   checking: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (19ms)
2023-09-23 13:59:37.897 [info] (87036) [BG(1)] analyzing: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (22ms)
2023-09-23 13:59:37.898 [info] (87036) [BG(1)] indexing: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py [found 1] (0ms)
2023-09-23 13:59:37.907 [info] (87036) [FG] parsing: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (1ms)
2023-09-23 13:59:37.907 [info] (87036) [FG] binding: /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (0ms)
2023-09-23 13:59:38.069 [info] (87036) [BG(1)] getSemanticTokens delta previousResultId:1695502767600 at /Users/euan/git/masters-tavern/masters_tavern/models/identity.py (1ms)

Thanks for the issue. Can you share your code or an example that reproduces the problem? With a trivial repro, I can't seem to reproduce:

My guess is that we think masters_tavern is a 3rd party import for some reason.

Sorry for the delay in responding. I've made a smallest repro for the behaviour here.

My guess is that we think masters_tavern is a 3rd party import for some reason.

That's my suspicion too, a lot of my initial searching was in the vein of pylance local "third party" import wrong. I thought it might be the re-exporting in some of the __init__.py files, but with the minimum repro having all __init__.py files empty it seems like I was wrong there.
In case it helps, for additional context I'm running on a 2021 Macbook Pro with an M1 Pro chip with OS version Ventura 13.5.2.

Thanks, that example lets me reproduce the problem.
I don't think this is very high priority at the moment though, so it will probably be a while before we get to fixing it.

This issue has been fixed in prerelease version 2024.2.100, which we've just released. You can find the changelog here: CHANGELOG.md


Fix the link to the Code of Conduct

Or we could rename the file so it has the same name as it has in ASP.Net Core repository.

Thanks. This format matches other repos in the Microsoft org so I think we'll keep it that way.


Add support for py_modules_only compilation

What does this PR do?
Create support handling the case of compiling packages containing py_modules only. Currently, Nuitka assumes that a main package always exists and runs into an exception when it is not the case.
Why was it initiated? Any relevant Issues?
Issue #479
GSoC2019
PR Checklist

[x] Correct base branch selected? develop for new features and bug fixes too.
If it's part of a hotfix, it will be moved to master during it.
[x] All tests still pass. Check the developer manual about Running the Tests.
There are Travis tests that cover the most important things however, and you
are welcome to rely on those, but they might not cover enough.
[ ] Ideally new features or fixed regressions ought to be covered via new tests.
[ ] Ideally new or changed features have documentation updates.

@kayhayen Need some pointers here
With the changes, python run_all.py only py_modules_only runs into the following exception:
Nuitka:WARNING:Recursed to module 'py_modules_only' at 'C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only\build\lib\py_modules_only.py' twice.
Nuitka:INFO:Included compiled module 'py_modules_only'.
error: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Tommy\\Nuitka\\tests\\distutils\\py_modules_only\\build\\lib\\py_modules_only'
Creating virtualenv for quick test:
Traceback (most recent call last):
  File "run_all.py", line 257, in <module>
    main()
  File "run_all.py", line 169, in main
    commands=['cd "%s"' % case_dir, "python setup.py bdist_nuitka"]
  File "C:\Users\Tommy\Nuitka\nuitka\tools\testing\Virtualenv.py", line 51, in runCommand
    assert os.system(command) == 0, command
AssertionError: call scripts\activate.bat && cd "C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only" && python setup.py bdist_nuitka

It seems that py_modules_only is assumed to be a directory? How should I approach this

Well, I recommend you trace the command before executing it.
Telling Nuitka that a module should be included as a package, is bound for disaster. No wonder, it's not found, it's a file and not a directory.
On modules, use --include-module on packages use include-package, do not fix them. Probably the error messages from Nuitka are not good or not there? Not sure how robust Nuitka is against your errors there.
Yours,
Kay

@kayhayen I fixed it to use only --include-module on modules. However I am still getting the same error.
Traceback:
Nuitka:WARNING:Recursed to module 'py_modules_only' at 'C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only\build\lib\py_modules_only.py' twice.
Nuitka:INFO:Included compiled module 'py_modules_only'.
error: [WinError 3] The system cannot find the path specified: 'C:\\Users\\Tommy\\Nuitka\\tests\\distutils\\py_modules_only\\build\\lib\\py_modules_only'
Creating virtualenv for quick test:
Traceback (most recent call last):
  File "run_all.py", line 257, in <module>
    main()
  File "run_all.py", line 169, in main
    commands=['cd "%s"' % case_dir, "python setup.py bdist_nuitka"]
  File "C:\Users\Tommy\Nuitka\nuitka\tools\testing\Virtualenv.py", line 51, in runCommand
    assert os.system(command) == 0, command
AssertionError: call scripts\activate.bat && cd "C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only" && python setup.py bdist_nuitka

Can you show the resulting Nuitka command then?

C:\Users\Tommy\Nuitka\tests\distutils\venv_nuitka\Scripts\python.exe -m nuitka --module --plugin-enable=pylint-warnings --output-dir=C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only\build\lib --nofollow-import-to=*.tests --show-modules --remove-output --include-module=py_modules_only C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only\build\lib\py_modules_only.py

The --include-module on the main module is not needed, unlike with packages, which will not automatically be fully included, but only top module, there is no extra benefit from that.
I cannot decipher the error message, it might not be from Nuitka, is it? I guess it is true:

error: [WinError 3] The system cannot find the path specified: 'C:\Users\Tommy\Nuitka\tests\distutils\py_modules_only\build\lib\py_modules_only

However, what is trying to use it?

@kayhayen
Made changes and replied to your comments above, please review :). The script now builds distutils/py_modules_only without error, but needs to be tweaked to support the case of both package and module I think

Instead of printing values at crash time, your go to solution should be to consider the callstack and see what it puts there. Your imaginary call stack, your usage of findModule is not the only one. You just encountered a regression of Nuitka develop itself. Please find it and make it a separate PR "fixup_import_module".
Seems we have no coverage in testing for that option until your test.
Yours,
Kay

@JorjMcKie The regression I just noted, should also affect your hinter maybe? Does it not use --include-module command line, or is it purely doing its stuff via plugin calls.

The hinted-mods.py user-plugin does this to adjust Nuitka compilation parameters:

directly append entries to ´´options.options.plugins_enabled``
directly append entries to options.recurse_modules

So it does not (yet) touch  options.include_modules or options.include_packages.

Ok, good to know, then it is not affected by the regression, and I need to panic rebase you :)

Anytime, just say when 😊
Von meinem iPhone gesendet

Please find it and make it a separate PR "fixup_import_module".

#487 created

Good job, I really like how this converged to unify the two cases.


[3.0.0] Publisher UI to configure maximum TPS for backends

The UI to set max backend throoughput for production and sandbox endpoints has to be implemented under configure tab.

Fixed with https://github.com/wso2/carbon-apimgt/pull/7151


Add support for logical data types

Is your feature request related to a problem? Please describe.
One limitation with the current DataType.check() method is that it only validates the data type, without access to the data itself. I've ran into this when implementing decimal and date dtypes, which is why I've been holding back my PR.
I'm quoting the documenation of the visions lib (a data type framework):

Physical types represent the actual, underlying representation of the data.
Logical types represent the abstracted understanding of that data.

According to those definitions, pandera DataTypes represent physical types, i.e. have a 1-1 relationship with pandas dtypes.
When I designed DataType, my position was that Check should be used if data is necessary to validate. However, I now think there are cases where it makes sense for DataType.check to receive data as well:

Dtypes unofficially supported by pandas: date, decimal. Those are understood by Pyarrow, which is especially useful when writing to Parquet. In theory, the object  can contain any types, but we'd need to look at the data to validate. Another example is the new PydanticModel introduced in #779
Logical types: IP, URLs, Paths, etc. Technically they can be emulated by a function that returns a pandera.Column. I find it unintuitive, and it would be awkward to introduce new column types in the public api. Moreover, coercion is not available with this technique, even if I don't have a concrete use-case to present.
e.g.:

def IPColumn(**kwargs:Any) -> pa.Column:
  """Re-usable helper to check columns containing IP addresses.""
  checks = kwargs["checks"] or []
  checks.append(pa.Check(is_valid_ip)) # assume we have is_ip_valid function
  kwargs["checks"] = checks
  return pa.Column(pa.STRING, **kwargs)

Describe the solution you'd like
We could add an optional argument data_container -> def check(self, pandera_dtype: DataType, data_container: Optional[Any]=None) -> bool. When schema validation takes place we always have access to the column data, it would be easy to pass it to check and let the DataType use it if needed. Current data types will simply ignore the new argument.

Will do !


Tests for casts in JS fail due to differing JS/JVM semantics

Tests such as TypeableTests.testTuples fail on JS as it is a dynamic language and tries to cast anything it can. These types of tests need to be changed on JS only to return false (i.e. they are not empty).
See Semantics of Scala.js

I guess I'm not surprised that there are issues with Typeable given that it uses reflection.
But surely there's an alternative runtime typing mechanism which could be used here instead? I would be happy for the Scala/JS branch to use that as an alternative to JVM reflection in the implementation of Typeable.

Makes sense, and I shall investigate accordingly.

Interestingly, this might make Typeable into a portable reflection-replacement.

Wasn't this one fixed a while back?

I'm not clear what happened here ... was Scala.js friendly implementation of Typeable added to the Scala.js branches in the end?

Fixed in 2e1a48ef5826e9df3b0aff66ea0d84341450a4a0.


app crashes on startup: no such table: connection

when start the app I restart the app with "heroku ps:restart", I get the error:

app/web.1: sqlalchemy.exc.OperationalError: (OperationalError) no such table: connection u'SELECT connection.conn_id AS connection_conn_id \nFROM connection GROUP BY connection.conn_id' ()

I can see that "airflow initdb" created the sqlite table in: /app/airflow/airflow.db, so not sure why it's failing.

This means that the PYTHONPATH env variable didn't get setup correctly.
Can you do heroku run bash and then echo $PYTHONPATH to see what it was set as?
Alternatively, I created this repo that has a Heroku Button if you don't want to use docker => https://github.com/neovintage/heroku-airflow

~ $ echo $PYTHONPATH
/app/user/airflow_login::/app/user

After playing with the heroku button version, I think the issue here was that AIRFLOW_HOME wasnt set so it's not point to the right airflow.cfg.  This shouldn't be using sqlite which is the default, it should use postgres.  It is also could be something with the DATABASE_URL var itself but I havent looked at it yet.

I fixed this by doing what @bryan0 suggested. After I had to run $airflow initdb and the app boots fine now.

@jetaggart did you fix it by setting AIRFLOW_HOME. If so what and where did you set it. The original repo had it set in the app.json but not inside Heroku.
app.json
...
"env": {
"AIRFLOW_HOME": {
"description": "Where all the information for airflow lives",
"value": "/app/user"
},
....

@jetaggart @neovintage I added the AIRFLOW_HOME as an environment variable with value of '/app/user'. That pointed to the right configue and I was able to initdb and see postgres spin up. However the app is still crashing.
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "User" in the registry of this declarative base. Please use a fully module-qualified path.
Any recomendations? Thank you.

Yeah, I'm seeing the same error. Here's the full backtrace:
Starting process with command `cd /app/user && airflow webserver -p 25107`
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
2016-04-29 18:59:09,797 - root - INFO - Filling up the DagBag from /app/user/dags
Traceback (most recent call last):
  File "/app/.heroku/python/bin/airflow", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/app/user/src/airflow/airflow/bin/airflow", line 10, in <module>
    args.func(args)
  File "/app/user/src/airflow/airflow/bin/cli.py", line 252, in webserver
    from airflow.www.app import app
  File "/app/user/src/airflow/airflow/www/app.py", line 1646, in <module>
    mv = JobModelView(jobs.BaseJob, Session, name="Jobs", category="Browse")
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 291, in __init__
    menu_icon_value=menu_icon_value)
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/model/base.py", line 587, in __init__
    self._refresh_cache()
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/model/base.py", line 663, in _refresh_cache
    self._list_columns = self.get_list_columns()
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/model/base.py", line 743, in get_list_columns
    columns = self.scaffold_list_columns()
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 435, in scaffold_list_columns
    for p in self._get_model_iterator():
  File "/app/.heroku/python/lib/python2.7/site-packages/flask_admin/contrib/sqla/view.py", line 313, in _get_model_iterator
    return model._sa_class_manager.mapper.iterate_properties
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line 1805, in iterate_properties
    configure_mappers()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line 2631, in configure_mappers
    mapper._post_configure_properties()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/mapper.py", line 1696, in _post_configure_properties
    prop.init()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/interfaces.py", line 144, in init
    self.do_init()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py", line 1550, in do_init
    self._process_dependent_arguments()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py", line 1606, in _process_dependent_arguments
    self.target = self.mapper.mapped_table
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 725, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/orm/relationships.py", line 1523, in mapper
    argument = self.argument()
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py", line 271, in __call__
    x = eval(self.arg, globals(), self._dict)
  File "<string>", line 1, in <module>
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/util/_collections.py", line 680, in __missing__
    self[key] = val = self.creator(key)
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py", line 252, in _access_cls
    return _determine_container(key, cls._decl_class_registry[key])
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py", line 236, in _determine_container
    value = value.attempt_get([], key)
  File "/app/.heroku/python/lib/python2.7/site-packages/sqlalchemy/ext/declarative/clsregistry.py", line 89, in attempt_get
    (".".join(path + [key]))
sqlalchemy.exc.InvalidRequestError: Multiple classes found for path "User" in the registry of this declarative base. Please use a fully module-qualified path.
Process exited with status 1


Loading screen can potentially get stuck

The problem

User expected behaviour:

Umbrel gets through loading and then flags any errors once in the dashboard

Actual behaviour:

Loading screen gets stuck indefinitely at 80%

Troubleshooting
I found this from accidentally (I think) breaking lnd on an umbrel-dev deploy. As an un-related issue, my lnd docker image hadn't started and wasn't currently running in the environment

On trying to load the loading screen, I ended up stuck at 80% with the vue app polling umbrel every 2 secs and producing the following logs

I expect it's because the following error is being thrown in the lnd status check and I guess my questions are:

for the frontend is this an issue, or are you all expecting lnd to always be setup correctly?
can this likely happen if another error causes the line below to be thrown (same with other status checks)?
if this can legitimately happen, what should the desired behaviour be from a UX standpoint?

https://github.com/getumbrel/umbrel-middleware/blob/fb870f493276f27721e164f010ebf01563ca6fb1/logic/lightning.js#L770

This looks like an umbrel-dev issue

This looks like an umbrel-dev issue

Underlying lnd error is almost certainly an umbrel-dev issue
I like how that one failure gave a nice edge case for testing the dashboard robustness though in an unrelated way. I'd say this issue is probably edge enough to slap it with a low-priority label and leave it for now until it either becomes a problem or stays undiscovered for long enough that it's no longer an issue.

Thanks for reporting this, @vindard! Great observations overall.
The dashboard calculates the loading progress by verifying if all the services are up and running (here's the logic).
Now:

for the frontend is this an issue, or are you all expecting lnd to always be setup correctly?

Yes, it's an issue, although we're also expecting lnd to be always set up correctly as both the lnd version and lnd.conf are deterministically installed. But in cases such as database corruption, or network failure, or something totally unexpected such as in your case, it's an issue because then we have no way to notify the user what went wrong.

can this likely happen if another error causes the line below to be thrown (same with other status checks)?

Yes, looks like that's where it is thrown, or else it wouldn't error out in umbrel-middleware's logs. I'm not sure if it's due to another problem though as both could be related to the same issue of lnd not starting up due to an "invalid argument".

if this can legitimately happen, what should the desired behavior be from a UX standpoint?

Good question! Since we can't login to the dashboard with any of the services not being up, a better UX would be to replace the progress bar with an error that shows what went wrong as we do not expect the users to be manually going through their logs.
Although, for the time being (beta), we can add a timeout on the loading screen that shows a tip on the likes of Hmm... it's taking unusually long to start Umbrel. You can check out the logs at http://umbrel.local/logs (where we currently tail /var/log/syslog).

Nice, yea that final suggestion sounds ideal for now, also maybe with some text in general on the different loading steps (once you guys aren't reticulating splines 😂 )?
I saw there was some commented code in the Vue app I think for that progress text already

Closing this now since we added the loading text :)


What will happen if I change the backbone network into MobileNet-v2 with FPN?

Hi,
I want to try to change the backbone network into MobileNet-v2 with FPN. Is there any suggestions? THX!!!

That would be a good test, though I will warn that we've tried VGG16+FPN before and got sub-par performance (though maybe that's to be expected). If you want to change out the backbone head over to backbone.py and check out how I implemented Resnet and VGG. If you need any help, let me know.

@dbolya Hi, I have done a preliminary test on my own simple dataset. And the experiment result made me feel excited~! With the help of MobileNet-v2, I successfully reduced the model size from over 100 MB to 31 MB, but without a significant performance loss on my own dataset. Next, i will try it on MS-COCO dataset, and i hope for the fantastic performance^_^

Sounds good! If it ends up working well, I might just make an official MobileNet-v2 version for YOLACTv1.1. That seems like it would be nice to have.

@dbolya Hi, could you show me your train log file of resnet&coco?

This is my recently train log with mobilenetv2&coco...

[ 16]   59810 || B: 2.489 | C: 3.295 | M: 2.861 | S: 0.897 | T: 9.542 || ETA: 19 days, 23:21:26 || timer: 2.437
[ 16]   59820 || B: 2.505 | C: 3.321 | M: 2.840 | S: 0.898 | T: 9.565 || ETA: 19 days, 22:54:13 || timer: 2.125
[ 16]   59830 || B: 2.500 | C: 3.299 | M: 2.802 | S: 0.910 | T: 9.511 || ETA: 19 days, 22:34:59 || timer: 1.966
[ 16]   59840 || B: 2.504 | C: 3.271 | M: 2.787 | S: 0.900 | T: 9.461 || ETA: 19 days, 22:51:36 || timer: 2.071
[ 16]   59850 || B: 2.480 | C: 3.282 | M: 2.751 | S: 0.881 | T: 9.394 || ETA: 19 days, 22:43:14 || timer: 2.415
[ 16]   59860 || B: 2.462 | C: 3.247 | M: 2.752 | S: 0.883 | T: 9.344 || ETA: 19 days, 22:27:50 || timer: 2.144
[ 16]   59870 || B: 2.460 | C: 3.247 | M: 2.726 | S: 0.881 | T: 9.315 || ETA: 19 days, 22:22:21 || timer: 2.091
[ 16]   59880 || B: 2.489 | C: 3.290 | M: 2.696 | S: 0.891 | T: 9.366 || ETA: 19 days, 22:38:05 || timer: 2.101
[ 16]   59890 || B: 2.478 | C: 3.300 | M: 2.712 | S: 0.892 | T: 9.382 || ETA: 19 days, 22:48:17 || timer: 2.301
[ 16]   59900 || B: 2.485 | C: 3.299 | M: 2.711 | S: 0.889 | T: 9.384 || ETA: 19 days, 22:53:21 || timer: 2.046
[ 16]   59910 || B: 2.514 | C: 3.316 | M: 2.724 | S: 0.880 | T: 9.435 || ETA: 19 days, 23:27:14 || timer: 1.973
[ 16]   59920 || B: 2.505 | C: 3.308 | M: 2.721 | S: 0.877 | T: 9.410 || ETA: 19 days, 23:25:19 || timer: 2.257
[ 16]   59930 || B: 2.496 | C: 3.290 | M: 2.710 | S: 0.863 | T: 9.360 || ETA: 19 days, 23:15:29 || timer: 2.061
[ 16]   59940 || B: 2.467 | C: 3.269 | M: 2.694 | S: 0.864 | T: 9.293 || ETA: 19 days, 22:23:27 || timer: 1.924
[ 16]   59950 || B: 2.473 | C: 3.249 | M: 2.702 | S: 0.868 | T: 9.292 || ETA: 19 days, 22:55:27 || timer: 2.374
[ 16]   59960 || B: 2.497 | C: 3.261 | M: 2.702 | S: 0.858 | T: 9.318 || ETA: 19 days, 23:18:36 || timer: 2.055
[ 16]   59970 || B: 2.532 | C: 3.274 | M: 2.726 | S: 0.862 | T: 9.393 || ETA: 19 days, 23:51:10 || timer: 2.111
[ 16]   59980 || B: 2.493 | C: 3.262 | M: 2.723 | S: 0.861 | T: 9.339 || ETA: 19 days, 23:32:02 || timer: 1.991
[ 16]   59990 || B: 2.500 | C: 3.253 | M: 2.718 | S: 0.860 | T: 9.331 || ETA: 19 days, 23:44:44 || timer: 3.396

The loss value looks like going down. But the convergence speed looks so slowly. And i got a poor evaluated result:

box |  0.78 |  2.17 |  1.76 |  1.39 |  1.08 |  0.75 |  0.43 |  0.19 |  0.07 |  0.01 |  0.00 |
box |  1.31 |  3.44 |  2.92 |  2.37 |  1.78 |  1.31 |  0.78 |  0.33 |  0.14 |  0.04 |  0.00 |
box |  1.47 |  3.88 |  3.23 |  2.54 |  2.05 |  1.47 |  0.82 |  0.44 |  0.21 |  0.04 |  0.00 |
box |  2.17 |  5.26 |  4.63 |  3.84 |  3.12 |  2.26 |  1.42 |  0.77 |  0.27 |  0.11 |  0.00 |
box |  2.41 |  5.79 |  5.10 |  4.26 |  3.46 |  2.54 |  1.74 |  0.82 |  0.34 |  0.06 |  0.00 |
box |  2.65 |  6.06 |  5.34 |  4.67 |  3.76 |  2.90 |  1.96 |  1.05 |  0.51 |  0.20 |  0.03 |
box |  2.82 |  6.62 |  5.94 |  4.82 |  4.05 |  2.94 |  2.04 |  1.17 |  0.53 |  0.12 |  0.00 |

mask |  0.80 |  1.91 |  1.61 |  1.35 |  1.09 |  0.83 |  0.58 |  0.38 |  0.21 |  0.07 |  0.00 |
mask |  1.44 |  3.13 |  2.71 |  2.28 |  1.95 |  1.62 |  1.20 |  0.81 |  0.55 |  0.12 |  0.01 |
mask |  1.73 |  3.73 |  3.28 |  2.81 |  2.35 |  1.88 |  1.43 |  0.96 |  0.57 |  0.26 |  0.03 |
mask |  2.38 |  4.86 |  4.39 |  3.80 |  3.27 |  2.65 |  2.09 |  1.49 |  0.95 |  0.29 |  0.03 |
mask |  2.76 |  5.46 |  5.04 |  4.48 |  3.84 |  3.15 |  2.43 |  1.70 |  0.96 |  0.44 |  0.06 |
mask |  3.01 |  5.81 |  5.34 |  4.82 |  4.16 |  3.53 |  2.72 |  1.97 |  1.10 |  0.56 |  0.07 |
mask |  3.31 |  6.41 |  5.85 |  5.27 |  4.56 |  3.94 |  2.99 |  2.13 |  1.31 |  0.58 |  0.06 |

Hope to discuss more with you ^_^

Hmm with the mAP starting that low (< 1), it looks likes the behavior we get when we train from scratch. Are you using imagenet pretrained weights in your MobileNet implementation? If not, it'll take way longer to converge.
There's also a gotcha when loading pretrained weights because I had to set strict=False in the torch.load for loading backbone weights. Set that back to True if you are using pretrained weights but suspect it's not loading properly.

THX~!
I used the implementation and the pretrained weights from this repo:https://github.com/tonylins/pytorch-mobilenet-v2
I did the experiment follow these steps bellow:

Modify the network architecture by removing the final global average pooling layer and the final 1x1 conv layer;
Make a custom init_backbone() method in which i translated the node name and discarded the unuseful tensor;
Add a new config-dict to the 'config.py', and set some helpful parameters.

Here is my MobileNet-v2 implementation:
class MobileNetV2(nn.Module):
    def __init__(self):
        super(MobileNetV2, self).__init__()
        block = InvertedResidual
        input_channel = 32
        interverted_residual_setting = [
            # t, c, n, s
            [1, 16, 1, 1],
            [6, 24, 2, 2], # useful
            [6, 32, 3, 2], # useful
            [6, 64, 4, 2],
            [6, 96, 3, 1], # useful
            [6, 160, 3, 2],
            [6, 320, 1, 1], # useful
        ]

        # Select the output layers
        # idx-1 => 56 * 56 * 24
        # idx-2 => 28 * 28 * 32
        # idx-4 => 14 * 14 * 96
        # idx-6 => 7 * 7 * 320
        select_idxs = [1, 2, 4, 6]

        # building first layer
        input_channel = int(input_channel)
        self.conv2d = conv_bn(3, input_channel, 2)
        self.layers = nn.ModuleList()
        self.channels = []

        # building inverted residual blocks
        features = []
        for idx, (t, c, n, s) in enumerate(interverted_residual_setting):
            output_channel = int(c)

            for i in range(n):
                if i == 0:
                    features.append(block(input_channel, output_channel, s, expand_ratio=t))
                else:
                    features.append(block(input_channel, output_channel, 1, expand_ratio=t))
                input_channel = output_channel

            if idx in select_idxs:
                self.layers.append(nn.Sequential(*features))
                self.channels.append(output_channel)
                features = []

        self.backbone_modules = [m for m in self.modules() if isinstance(m, nn.Conv2d)]

        # random initial
        self._initialize_weights()

    def forward(self, x):
        x = self.conv2d(x)

        outs = []
        for layer in self.layers:
            x = layer(x)
            outs.append(x)

        return tuple(outs)

    def _initialize_weights(self):
        modules = self.modules()
        for m in modules:
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()

    def init_backbone(self, path):
        """ Initializes the backbone weights for training. """
        state_dict = torch.load(path, map_location='cpu')

        # Replace featuresXXX -> layers.x.xx(/conv2d.x) etc.
        keys = list(state_dict)
        for key in keys:
            if key.startswith('features'):
                idx = int(key.split('.')[1])
                if idx <= 17:
                    if idx == 0:
                        new_key = 'conv2d.' + key[11:]
                    elif (idx >= 1 and idx <= 3):
                        new_key = "layers.0." + str(idx - 1) + key[10:]
                    elif (idx >= 4 and idx <=6):
                        new_key = "layers.1." + str(idx - 4) + key[10:]
                    elif (idx >= 7 and idx <= 13):
                        if idx <= 9:
                            new_key = "layers.2." + str(idx - 7) + key[10:]
                        else:
                            new_key = "layers.2." + str(idx - 7) + key[11:]
                    else:
                        new_key = "layers.3." + str(idx - 14) + key[11:]

                    state_dict[new_key] = state_dict.pop(key)
                else:
                    state_dict.pop(key)
            else:
                state_dict.pop(key)

        # Note: Using strict=False is berry scary. Triple check this.nn.ModuleList()
        self.load_state_dict(state_dict, strict=False)

And there is the corresponding config:
mobilenetv2_backbone = resnet50_backbone.copy({
    'name': 'MobileNetV2',
    'path': 'xxx.pth',
    'type': MobileNetV2,
    'args': (),
    # 'transform': resnet_transform,

    'selected_layers': [1, 2, 3],
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,
    'use_pixel_scales': False,
})
yolact_mobilenetv2_coco_config = yolact_resnet50_config.copy({
    'name': 'yolact_mobilenetv2_coco',

    'backbone': mobilenetv2_backbone.copy(),

    'use_semantic_segmentation_loss': True
})

haha, i found the problem...Just after i commented...
When i do test on my own dataset, i removed some other anchors on purpose... and i forgot to recover them.

Hmm, can you still do a sanity check and change self.load_state_dict(state_dict, strict=False) to self.load_state_dict(state_dict, strict=True) and see if there's anything unexpected is missing? I think even with one anchor per layer, you shouldn't get < 1 mAP on the first validation epoch.

Thx, i'm just going to do it!

So,what's the accuracy and fps with mobilenet v2  now?@Wilbur529

@andeyeluguo
It has run almost 200 epochs now, and the mAP are 14.36(box) and 15.02(mask) respectively. But I found that the model does not converge completely, so keep waiting...

@Wilbur529  wonderful work on the modification on the network backbone . keep going !!!!!!!!!!

@Wilbur529  hi is it possible to share your mobilenet implementation py  ?? how good is the accuracy and fps compared to resenet 50,101

@abhigoku10 You could find the model and the accuracy in my previous answers, and i will do a benchmark test later. As far as I can see from the training log, i think it would be vary fast.

@Wilbur529  i was able to train the model using mobilenet architecture, thanks for sharing the reference implementation , when i tested it on images its able to detect with low accuracy but when i run it for a video the detection are not happening at all . Did u face the similar issue , should we make any changes with the code

@abhigoku10 Good job! I think it may not be a difficult problem. If you couldn't find the error code, why not write a for-loop upon your image inference function to simulate the video inference:)

@Wilbur529  yup that was my next step , but the video execution uses threading and mutiframe buffer functionality so just trying to check the feasibility on that , but did u face the same issues with the video

@abhigoku10 Sry i did not face this problem. Based on your description, i think the model has no mistake. So just pay more attention to the evalvideo method, and check the postprocess procedure, good luck for u~

@Wilbur529 sure i shall do it , how many iterations you had trained ur mobilenet model , since i trained with 2 lakh iterations the results on the validation set is not that great though the fps is high

@abhigoku10 I let it ran on only one single GPU, so it needed long time for training. At 200 epochs,  the mAP are 14.36(box) and 15.02(mask).

@Wilbur529  thanks for the info i shall run it to 200 epochs and share my mAP, anyother architecture u tried besides mobilenet

@abhigoku10 EfficientNets and MobileNet-v3 is coming~~~

@Wilbur529  cool any plans on sharing the reference code

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

@Wilbur529  thanks for the update eagers waiting for this

I am reading both of their paper, and i haven't found a suitable pytorch implementation of them~ Once I succeed in my experiment, i will show the modified parts.

Looking for update~there is a [reference] that may be helpful

EfficientNet is really efficient! I did some experiments:

@Wilbur529 can you share the reference code for the efficient net so that i can test it on different data , what is the fps and accuracy u achieved

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).

EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

@Wilbur529 wonderful experimentation and anlaysis , would you be able to share the yolact-efficient code , i would be able to perform certain more experiments , since effcient-b0 model size is an interesting one
THnaks in advance

@Wilbur529 @dbolya   i have trained the model with efficient b0 and b4 here is my following observations .

Efficient b0 is of model size of 42MB its mAP is similar to Resnet 18 which is of size 72MB but the performance of b0 for multiframe=2 is 20 fps but for resent18 with multiframe=2 is 22 fps for a resolution of 1280x720, which is strange since the model size and params was less but we getting less fps compared to resnet18
Similary b4 is of model size of 96MB if of 16fps for multiframe=2 but Resnet 50 which is of size 125MB is of 18fps  for multiframe=2
I am testing on 2080TI gpu system . any suggestions on this behaviour

@abhigoku10 I may suggest you only measure the network inference time:)

@dbolya i shall check up my implementation once again , i agree its not directly proportional that fewer params higher accuracy but is it not higher fps

What does 'selected_layers': [1, 2, 3] mean?

@zhangbaoj I replied in your new issue.

@abhigoku10 can you share your yolact-efficient code?

@Wilbur529 @abhigoku10 can you help me with the efficient backbone? i trained efficientnet-b3 but i only have 30.05 box and 28.13 mask (using pretrained weights). I think my problem is in the config file, can you share that with me pls?
pd: i'm using resnet50 config copy and [3,4,5] selected layers.
Thx :)

@dbolya Any word regarding integrating a new backbone into yolact?

@deepseek I didn't intend to add any new backbones to the master branch, but I guess if enough people want mobile net, I could try.

@dbolya i have an efficient net cod but i wanted to review it is it possbile to review it before merging into the branch @Jackerz312  and @deepseek  once is get it reviewed i can share it

@abhigoku10 Can you make a pull request for it? Then we can all review it.

@abhigoku10 @dbolya EfficientNet is really efficient!!! I used the implementation and pretrained weights from EfficientNet-PyTorch. To validate the availability, i did two experiments:

EfficientNet-B0 has similar Top-5 Acc.(93.2%) with ResNet-50(93.0%) on ImageNet. When i change the backbone of YOLACT to EfficientNet-B0, the convergence rate and mAP between them almost is same. But the total model sizes are YOLACT550-EfficientNet-B0(47MB) and YOLACT550-ResNet-50(120MB).
EfficientNet-B4 has similar Top-5 Acc.(96.3%) with SENet(96.2%) on ImageNet, and its parameters size(19M) is less than ResNet-50(26M). When i change the backbone of YOLACT to EfficientNet-B4, i found it may achieve a better performance beyond the previous network. After 55 epochs training, now it have achieved 30.95 mAP on COCO test-dev, and it seems not converge completely. It is important to note that the YOLACT550-EfficientNet-B4 model size is only 101MB.

hi, Wilbur529，can you please share your efficiennet code and the config file, i tried to used the efficiennet-B5, train from scratch and pretrained, after 55 epochs training , i can only get about 19 map. how many layers have you used in the FPN, can you share the layers that you use?

@Wilbur529
I meet the warning " Warning: Moving average ignored a value of inf " During the  training , when I change  the backbone of resnet to mobilenetv2 . Did you met this warning ?

@xscjun I have a patch coming as a result of #222 that might fix that issue for you. Try it out.

@Wilbur529 ,
Can you please share your EfficientNet backbone?
How can i use it in me network?

What the FPS did you get?
On which GPU?

@IMFinethankyou , can you share you'r config and backbone file?
I don't know how to do it...

hi, sdimantsd, you can check this https://github.com/toandaominh1997/EfficientDet.Pytorch, my backbone file is from this repo. and the config is much like that of the resnet-101 version, except that you need to be careful about the anchors setting.
------------------ 原始邮件 ------------------
发件人<EMAIL_ADDRESS>发送时间: 2019年12月12日(星期四) 晚上9:23
收件人<EMAIL_ADDRESS>抄送: "Dracula"<1010502718<EMAIL_ADDRESS>主题: Re: [dbolya/yolact] What will happen if I change the backbone network into MobileNet-v2 with FPN? (#27)
@IMFinethankyou , can you share you'r config and backbone file?
I don't know how to do it...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

@IMFinethankyou , i tried this one, but YOLACT need more functions/fields like:
add_layer()
layers

@sdimantsd , add_layer() is used in the backbone construction process, not neccessary in EfficientNet, and the #layers# only used in the load_weights() function for compatability of different  version of backbone weights,  also not necessary for EfficientNet

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?
Thanx!!

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?
Thanx!!
26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0  41FPS

Thanks!
And what about the anchors? How can i know the best settings?

@IMFinethankyou  i performed the same thing but i had got same accuracy for B4 , as @sdimantsd  was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',
# 'fpn': None,
# Dataset stuff
'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,
# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,

'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})
(i copyed from another backbone, im not shur if  it's good)
When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels
If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])
Can you please help me?

@IMFinethankyou, I don't know what im doing wrong, but i steel can't manage EfficientNet to work as backbone...
I used it as backbone the following config:
efficient_net_config = yolact_base_config.copy({
'name': 'efficient_net',
'fpn': None,
Dataset stuff
'dataset': ws_datasetV1,
'num_classes': len(ws_datasetV1.class_names) + 1,
# Training params
'lr_steps': (280000, 600000, 700000, 750000, 850000, 1000000, 1100000, 1200000, 1300000, 1350000, 1400000, 1450000),
'max_iter': 800000000,

'masks_to_train': 300,
'max_size': 700,

'backbone': efficient_net_backbone.copy({
    'selected_layers': list(range(2, 5)),

    'pred_scales': yolact_base_config.backbone.pred_scales,
    'pred_aspect_ratios': yolact_base_config.backbone.pred_aspect_ratios,
    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': True,  # This is for backward compatability with a bug
}),

})
(i copyed from another backbone, im not shur if it's good)
When i try to train i get the following error:
AttributeError: 'EfficientNetBackbone' object has no attribute 'channels'
in yolact.py line 426:
src_channels = self.backbone.channels
If i set backvone.channels to []
It crash in line 430:
self.fpn = FPN([src_channels[i] for i in self.selected_layers])
Can you please help me?

backbone.channels is a list, contains the channels of the output feature map
for instance: you backbone output a tuple of 5 feature maps:
torch.Size([1, 64, 67, 67])
torch.Size([1, 128, 33, 33])
torch.Size([1, 176, 16, 16])
torch.Size([1, 304, 8, 8])
torch.Size([1, 512, 4, 4])
then the channels should be [64, 128, 176, 304, 512]
backbone.channels is used latter in the FPN module

@IMFinethankyou i performed the same thing but i had got same accuracy for B4 , as @sdimantsd was asking did u re-generate the anchors ?? i used the mobile net anchors . Can you share what changes you have made in the config file

I borrowed the config from the base-config,  no special changes. the only change i made is set USE_SQUARE_ANCHORS to False.

@IMFinethankyou, thanks, i check it now.
How can i know the backbone output?

also, init_backbone() is not exist :-(

I don't know how you did it, Also backbone_modules is missing...

@IMFinethankyou Can you help me plz?

@IMFinethankyou Can you help me plz?

@sdimantsd, you should modify the efficientnet according to the backbone.py, so it can fit into the YOLACT. my connection to github is rediculously slow.

@dbolya Hello thanks for your amazing work. When I change the backbone to MobileNet by the following the above code which is provided by @Wilbur529, I got this error:

RuntimeError: copy_if failed to synchronize: device-side assert triggered
And sometimes I got Nan and inf in the loss However I am working with the latest version with has this problem was solved by #222 (Note: I got it only when changing the backbone to mobilenet)

@Wilbur529 @abhigoku10 Thank you for providing us by a reference code for MobilnetV2 backbone but unfortunately I doesn't work for me as I mentioned above so did you face the above problems and could help?
Are there any changes than the above provided code?
Thanks in advance.

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.
But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

Hello,
@Jackerz312 @Wilbur529 @abhigoku10
could you provide us with your implementation to Efficient-net backbone as It seems I couldn't port it in right way I followed this implementation https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py
I have a couple of questions @dbolya which may help me in porting it in the right manner
1- How should I set this values? will they be affected by changing the backbone?
'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,

2- Regarding the output I set it like that self.select_idxs = [1, 2, 4, 6]
3- should I take care of any other configuration while porting a new backbone?
For your information currently the loss is stack at certain high value and the mAP is zero
Thanks in advance.

@IMFinethankyou
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 529, in
train()
File "train.py", line 319, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 256, but got 1-dimensional input of size [3] instead
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

I solved the above issue by lowering the learning rate to 1e-4 instead of 1e-3.
I mentioned the solution to help others who may face the same issue.
But I faced another problem now, the mAP on the validation dataset (I used a custom dataset) as shown in the below figure

Note that: The training is working fine when I used the base backbone, the problem appears only when I changed the backbone to mobilenet.

The gap between box mAP and mask mAP is not abnormal. You may find the answer in the author'
s new papar YOLACT++.

@IMFinethankyou @Jackerz312 @Wilbur529 @abhigoku10
I use the efficientnet as the backbone show this errors :
Traceback (most recent call last):
File "train.py", line 532, in
train()
File "train.py", line 322, in train
losses = net(datum)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
raise output
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
output = module(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "train.py", line 147, in forward
preds = self.net(images)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/datasets/yolact-master/yolact.py", line 600, in forward
proto_out = self.proto_net(proto_x)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 256 16 3 3, expected input[2, 40, 69, 69] to have 16 channels, but got 40 channels instead
the config as follow:
yolact_EfficientNet_mydataset_config = yolact_base_config.copy({
'name': 'yolact_EfficientNet',
'dataset': my_custom_dataset,
'num_classes': len(my_custom_dataset.class_names) + 1,

'fpn': None,
'max_iter': 120000,
'lr_steps': (60000, 100000),

#'mask_proto_src': 0,

'backbone': EfficientNet_backbone.copy({
    'selected_layers': list(range(1, 4)),
    'pred_scales': [[24], [48], [96], [192], [384]],
    'pred_aspect_ratios': [[[1.414]]] * 5,

    'use_pixel_scales': True,
    'preapply_sqrt': False,
    'use_square_anchors': False,  # This is for backward compatability with a bug

}),

})
can you help me,please?
Can you please share your EfficientNet backbone code?
thks in advance

It seems like you did not change the channels num properly. You can check the first few layers in the protonet.

Hey @Wilbur529, Have you tried EfficientNet as a backbone? if yes, what performance and FPS have you achieved compare to ResNet50? Are you planning to share your EfficientNet backbone?
Also, I have a question about your MobileNet backbone that what is the intuition behind of selecting these layers  select_idxs = [1, 2, 4, 6] not other layers?
Thanks!

@Auth0rM0rgan i was able to implement efficient net as backbone the performance was lower compared to resnet for objects like pedistrains and the fps was higher than resent 50 , i just used the same pred_scales, ascpect ratios similar to mobile net not sure is that the right approach , i could not share the implementation due to firm polices , @Wilbur529 @dbolya can you suggest     'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,
'use_pixel_scales': False, is this the rigthvalues selected

------------------ 原始邮件 ------------------
发件人<EMAIL_ADDRESS>发送时间: 2020年2月18日(星期二) 晚上11:07
收件人<EMAIL_ADDRESS>抄送: "执着"<linshaodan66<EMAIL_ADDRESS>主题: Re: [dbolya/yolact] What will happen if I change the backbone network into MobileNet-v2 with FPN? (#27)
@Auth0rM0rgan i was able to implement efficient net as backbone the performance was lower compared to resnet for objects like pedistrains and the fps was higher than resent 50 , i just used the same pred_scales, ascpect ratios similar to mobile net not sure is that the right approach , i could not share the implementation due to firm polices , @Wilbur529 @dbolya can you suggest     'selected_layers': [1, 2, 3],
'pred_scales': [[24], [48], [96], [192], [384]],
'pred_aspect_ratios': [[[1.414]]] * 5,
'use_pixel_scales': False, is this the rigthvalues selected
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

@abhigoku10
i was able to implement efficient net as backbone，but after training to 1000 steps the  loss gradient cannot be reduced.i use efficient-b0 b3 b4, b4 of loss gradient connot be reduced,b0 and b3 of mAP is very low.
please help ！
Thanks  in advance！

@linshaodan  even i had the same behaviour i had trained for 6lakh iterations , but my mAP was low compared to resnet50 but fps was high

@abhigoku10
Yes, I got the same results as you said.
thank you.

Can you tell me what was the FPS on one of the "default" YOLACT backbone?

OK, i try it.
What do you mean by "anchors setting"? how do i need to change it?
What was the accuracy/map on you'r network?
What was the FPS? on what GPU?
Can you tell me what was the FPS on one of the "default" YOLACT backbone?
Thanx!!

26.8 mAP for B0, 29.6mAP for B3.
GTX 1080Ti, base YOLACT 37 FPS
EfficientNet-B0 41FPS

could you tell me the mAP for val set and test-dev?

@IMFinethankyou Hi,
I modify EfficientDet as EfficientNet backbone,
but I met this:

during training with efficientnet-b0 backbone.
And the loss doesn't reduce after iteration 1000.
The config as shown below:
efficientnet_backbone = resnet101_backbone.copy({
'name': 'EfficientNet',
'path': 'xxx.pth',
'type': EfficientNetBackbone,
'transform': efficientnet_transform,
'selected_layers': list(range(1, 4)),
})
'backbone': efficientnet_backbone.copy({
'selected_layers': list(range(1, 4)),
'use_pixel_scales': True,
'preapply_sqrt': False,
'use_square_anchors': False,
'pred_aspect_ratios': [[[1.414]]] * 5,
'pred_scales': [[24], [48], [96], [192], [384]],
}),
I don't know what goes wrong.
Can you share your config and backbone file or give some suggestions?
thks.

I made some study and changed the backbone to EfficientNet, you can have my source code at SpaceView/Yolact_EfficientNet, Please go there and try.
The source code uses EfficientNet-b0, but you can change it to any sub-model easily.
I can say that the training speed is really much slower even than the Resnet50, and I searched that this is well noted by lots of done work. Currently I don't have enough time to fine-tune this model, not to say finish the training.

haha, i found the problem...Just after i commented... When i do test on my own dataset, i removed some other anchors on purpose... and i forgot to recover them.

did you remove anchors?  where to recover them?


BUG Fix _configure_database.php being ignored

Fixes #7590 and Fixes https://github.com/silverstripe/silverstripe-sqlite3/issues/38
Also deprecates global $databaseConfig for _configure_database.php (no longer necessary, but still supported).
SQLite3 fix at https://github.com/silverstripe/silverstripe-sqlite3/pull/37

@tractorcow This is good to merge on green. Just a few seemingly unrelated tests failing?

Ok, rebased... let's see if this goes green!

Ok, rebased... let's see if this goes green!

afraid not

Rebased and hopefully this will fix it.
I had to remove the serve dev-master constraint, and put in a numeric constraint instead.


Is there any interest in contributions to make the clientgen outputs C++17 compatible

I've been using this lib (which is great by the way :D ) to generate a server for a standalone project, and a little helper lib for a client program we intend to plug into it.
This client program is unfortunately restricted to ++17 due to another 3rd party lib which uses language features that were removed in 20.
I've had a play around with modifying the CMakeLists and have ended up with both a ++20 aware server compiling, as well as a test program using the clientgen outputs that is only ++17 aware.
Obviously having ++17 compatible clientgen code should improve the distributability of client code - but might put it at odds with the expected roadmap (or just make it harder to maintain).
Curious to see if that would be welcome addition or not - let me know.

I don't want to guarantee that clientgen or one of the runtime libraries the generated code links to won't eventually require C++20 (or later) in the future, but if/when there's a change that big, it'll probably trigger a 5.x version. I think it's safe to pursue this for the 4.x series. 👍🏼


OpenSSL::Test: add data_dir()

For test recipes that want to use the directory of the data directory
or a subdirectory thereof, rather than just individual files.

Merged.
708a6a17592865590344eca541cbfccd472d7b45 OpenSSL::Test: add data_dir()


support configurable containerd and runc url

Currently, only the versions of containerd and runc are configurable by picking the pre-cached deb package file, this makes tests with containerd and runc from other sources impossible, this change introduces new fields in NodeBootstrappingConfiguration, ContainerdPackageURL and RuncPackageURL compared to existing ContainerdVersion and RuncVersion . These changes also changed the CSE file to always install containerd or runc from the package URLs first if specified, and exit without trying other installation methods.

validated the change in standalone environment for URLs are specified and not.

sorry, caused some conflicts because we had to tweak containerd/runc install a bit for 1.5.9/1.4.12 upgrade. but I think it should make your life easier -- we always use the URL for containerd.
mostly lgtm if you can rebase the containerd install

@alexeldeib, I have tested my change in AKS, please take another look, Thanks.