WMDataMining fail to retrieve some data from McM (in testbed)

Hey folks, while scanning the reqmgr2 logs in cmsweb-testbed, I noticed this error is being raised every 15min
[01/Apr/2015:02:27:24]  INFO: Active Requests: Getting job information from https://cmsweb-testbed.cern.ch/couchdb/wmstats and https://cmsweb-testbed.cern.ch/couchdb/reqmgr_workload_cache. Please wait.
[01/Apr/2015:02:27:29]  INFO: Active Requests: Total 2115 requests retrieved

[01/Apr/2015:02:27:39] vocms0133.cern.ch <IP_ADDRESS> "GET /reqmgr2/data/about HTTP/1.1" 200 OK [data: 338 in 780 out 636656 us ] [auth: OK "" "" ] [ref: "" "ServerMonitor/2.0" ]
[01/Apr/2015:02:27:42]  ERROR: <type 'exceptions.ValueError'> getting history from McM for PREP ID task_SUS-Fall13wmLHE-00011. Unknown error.
[01/Apr/2015:02:27:43]  ERROR: <type 'exceptions.ValueError'> getting history from McM for PREP ID REDIGITEST-0001. Unknown error.
[01/Apr/2015:02:27:43]  Periodic Thread ERROR __builtins__.KeyError 'request_status'
[01/Apr/2015:02:27:43]   Traceback (most recent call last):
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/CherryPyPeriodicTask.py", line 74, in run
[01/Apr/2015:02:27:43]       self.taskFunc(self.config)
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/WMDataMining.py", line 30, in gatherActiveDataStats
[01/Apr/2015:02:27:43]       archived = False, log = cherrypy.log)
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/WMDataMining/Utils.py", line 229, in gatherWMDataMiningStats
[01/Apr/2015:02:27:43]       for status in requests[wf]['request_status']:
[01/Apr/2015:02:27:43]   KeyError: 'request_status'

I know 0 about this service, but it seems it fails to retrieve wf information from McM (based on the prepID) and then it fails to find that key.
BTW, I just found this one as well
[30/Mar/2015:02:30:22]  INFO: Active Requests: Total 2071 requests retrieved

[30/Mar/2015:02:30:34]  Periodic Thread ERROR __builtins__.KeyError 'inputdataset'
[30/Mar/2015:02:30:34]   Traceback (most recent call last):
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/CherryPyPeriodicTask.py", line 74, in run
[30/Mar/2015:02:30:34]       self.taskFunc(self.config)
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/WMDataMining.py", line 30, in gatherActiveDat
aStats
[30/Mar/2015:02:30:34]       archived = False, log = cherrypy.log)
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/WMDataMining/Utils.py", line 137, in gatherWMDataMiningStats
[30/Mar/2015:02:30:34]       inputdataset = requests[wf]['inputdataset']
[30/Mar/2015:02:30:34]   KeyError: 'inputdataset'

Thanks Alan. Let me look into this. It looks like I didn’t manage to trap every exception that could be thrown.
Eric

On Apr 1, 2015, at 10:19, Alan Malta Rodrigues<EMAIL_ADDRESS>wrote:
Hey folks, while scanning the reqmgr2 logs in cmsweb-testbed, I noticed this error is being raised every 15min
[01/Apr/2015:02:27:24]  INFO: Active Requests: Getting job information from https://cmsweb-testbed.cern.ch/couchdb/wmstats and https://cmsweb-testbed.cern.ch/couchdb/reqmgr_workload_cache. Please wait.
[01/Apr/2015:02:27:29]  INFO: Active Requests: Total 2115 requests retrieved
[01/Apr/2015:02:27:39] vocms0133.cern.ch <IP_ADDRESS> "GET /reqmgr2/data/about HTTP/1.1" 200 OK [data: 338 in 780 out 636656 us ] [auth: OK "" "" ] [ref: "" "ServerMonitor/2.0" ]
[01/Apr/2015:02:27:42]  ERROR: <type 'exceptions.ValueError'> getting history from McM for PREP ID task_SUS-Fall13wmLHE-00011. Unknown error.
[01/Apr/2015:02:27:43]  ERROR: <type 'exceptions.ValueError'> getting history from McM for PREP ID REDIGITEST-0001. Unknown error.
[01/Apr/2015:02:27:43]  Periodic Thread ERROR builtins.KeyError 'request_status'
[01/Apr/2015:02:27:43]   Traceback (most recent call last):
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/CherryPyPeriodicTask.py", line 74, in run
[01/Apr/2015:02:27:43]       self.taskFunc(self.config)
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/WMDataMining.py", line 30, in gatherActiveDataStats
[01/Apr/2015:02:27:43]       archived = False, log = cherrypy.log)
[01/Apr/2015:02:27:43]     File "/data/srv/beHG1504d/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre9/lib/python2.6/site-packages/WMCore/WMDataMining/Utils.py", line 229, in gatherWMDataMiningStats
[01/Apr/2015:02:27:43]       for status in requests[wf]['request_status']:
[01/Apr/2015:02:27:43]   KeyError: 'request_status'
I know 0 about this service, but it seems it fails to retrieve wf information from McM (based on the prepID) and then it fails to find that key.
BTW, I just found this one as well
[30/Mar/2015:02:30:22]  INFO: Active Requests: Total 2071 requests retrieved
[30/Mar/2015:02:30:34]  Periodic Thread ERROR builtins.KeyError 'inputdataset'
[30/Mar/2015:02:30:34]   Traceback (most recent call last):
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/CherryPyPeriodicTask.py", line 74, in run
[30/Mar/2015:02:30:34]       self.taskFunc(self.config)
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/ReqMgr/CherryPyThreads/WMDataMining.py", line 30, in gatherActiveDat
aStats
[30/Mar/2015:02:30:34]       archived = False, log = cherrypy.log)
[30/Mar/2015:02:30:34]     File "/data/srv/beHG1504c/sw.pre/slc6_amd64_gcc481/cms/reqmgr2/1.0.5.pre7/lib/python2.6/site-packages/WMCore/WMDataMining/Utils.py", line 137, in gatherWMDataMiningStats
[30/Mar/2015:02:30:34]       inputdataset = requests[wf]['inputdataset']
[30/Mar/2015:02:30:34]   KeyError: 'inputdataset'
—
Reply to this email directly or view it on GitHub https://github.com/dmwm/WMCore/issues/5746.

Eric W. Vaandering
Computing Physics Developer
CMS Computing Development Lead
Scientific Computing Division
Fermi National Accelerator Laboratory
630 962 7260 office
www.fnal.gov http://www.fnal.gov/

We think this is due to RegMgr2 missing some data. I've protected the code against it and then @ticoann  can see which workflow(s) have a problem.

BTW, nothing to do with McM, but it's good to see those errors are now being caught.


gorename: too many open files

gorename frequently aborts with the error "too many open files". It doesn't always happen. That then triggers a long cascade of failures of "undeclared name".
open /Users/anacrolix/src/go/src/net/ipsock.go: too many open files
open /Users/anacrolix/src/go/src/net/ipsock_posix.go: too many open files
/Users/anacrolix/src/go/src/net/sock_posix.go:20:2: undeclared name: netaddr

This seems relevant, it's the default on my system:
$ ulimit -a
open files                      (-n) 256

Thanks for the report.  I thought I had fixed this already.  Will take a look tomorrow.

I was on revision 473fd854f8276c0b22f17fb458aa8f1a0e2cf5f5 of go.tools

I was unable to reproduce the problem by setting ulimit -n; I only noticed problems when the limit was so low that it couldn't even get started.  But this CL should reduce the concurrency of file I/O:
https://go-review.googlesource.com/#/c/8421/
(The earlier fix I mentioned was for a different parallel I/O operation in the same tool.)
Could you try patching this change to see if it helps? Thanks.

I ran into this issue a lot last week when using errcheck, so I submitted CL #8475 as a fix. I didn't realize there was an open issue and a different fix until I saw the conflicting change listed in Gerrit.
On a project I work on, errcheck fails with this error the majority of the time using the master revision of go/loader. It does not fail with this error using the branch from CL 8421 or CL 8475 (although with CL 8421, it should still be possible to have this error occur in conf.FindPackage).

I agree that it's still possible in principle for the error to occur.   (It's an unfortunate consequence of ignoring the possibility of I/O failures in functions like (*go/build.Context) isDir.)  But I think the CL 8421 fix should eliminate it in practice; I suggest you revert CL #8475.
If the problem recurs, we'll revisit that change.

FWIW, I reproduced it running go/master db2e73faeb97f39653180080b0cc4c05501cbf94, with ulimit -n == 256. The renaming happened but a lot of "too many open files" errors were displayed.

I still get it on occasion with go1.5.

anacrolix@Matts-MacBook-Pro:~/tmp$ gorename -from '"github.com/anacrolix/torrent".torrent.DisplayName' -to displayName
While scanning Go workspace:
Package "github.com/mattn/go-gtk/example/expander": open /Users/anacrolix/gopath/src/github.com/mattn/go-gtk/example/expander/expander.go: too many open files.
Renamed 3 occurrences in 2 files in 1 package.

I also have this problem often, but not every time I run it.
go version go1.5.1 darwin/amd64


rustls-tls-native-roots does blocking IO during .build

repro
Cargo.toml
[package]
name = "test-rs"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "~0.12", default_features = false, features = ["rustls-tls-native-roots"] }

use reqwest::ClientBuilder;

fn main() {
    for _ in 0..10 {
        let start = std::time::Instant::now();
        ClientBuilder::new().build().unwrap();
        println!("built in: {:?}", start.elapsed());
    }
}

this produces the following flamegraph, which include loading certs std::fs.
Not sure what's a nice solution as wrapping the load in tokio::spawn_blocking would introduce a breaking change.

oops forgot the attach the flamegraph

Peeking in the source, yep, it does load a file from the disk. Thinking more about what exactly it's doing, that's not surprising. I'd recommend trying to only build one or a couple clients and sharing them.
I suppose one option is that you could load the native cert yourself manually (asynchronously), add it as a root certificate to the reqwest builder, and not use the rustls-tls-native-roots feature.

loading the certs manually certainly sounds like a good solution. Let me try that, thanks for the suggestion!


2.0.1 fails to build with libcint-5.0.0: error: typedef redefinition with different types ('struct CINTEnvVars' vs 'struct CINTEnvVars')

In file included from /wrkdirs/usr/ports/science/py-pyscf/work-py38/pyscf-2.0.1/pyscf/lib/gto/ft_ao.c:48:
/wrkdirs/usr/ports/science/py-pyscf/work-py38/pyscf-2.0.1/pyscf/lib/gto/ft_ao.h:86:3: error: typedef redefinition with different types ('struct CINTEnvVars' vs 'struct CINTEnvVars')
} CINTEnvVars;
  ^
/usr/local/include/cint.h:225:3: note: previous definition is here
} CINTEnvVars;
  ^
1 error generated.

I also ran into this issue trying to build PySCF 2.0.1 against libcint 5.0.0 in the development version of Fedora (rawhide).

https://github.com/pyscf/pyscf/issues/1126#issuecomment-989944491

I guess the structure defined in ft_ao.h can be commented out. @sunqm are there any other things need to be changed in pyscf?

Please keep libcint at v4.* . cint v5.* is working but pyscf as of 2.0.1 is not compatible with cint v5.*

Which PySCF will be compatible with libcint v5?

Next minor release maybe. The upgrade of libcint interface is coupled with some new features and PRs. Currently it still has problems to merge the new features. It makes the upgrade a little bit complicated

Fixed in PR #1384


Incorrect outline shown for object literal properties within a function argument

🔎 Search Terms
outline for object literals as function call argument
🕗 Version & Regression Information

⏯ Playground Link
No response
💻 Code
myFunction('foo', { foo: 1 });

🙁 Actual behavior
TS extension in vscode shows outline with a single element foo

🙂 Expected behavior
myFunction('foo', {foo: 1})

(at least not see foo as a separate outline-tree element
Additional information about the issue
No response

I would also expect the behavior in #30515 because there's a variable definition, which is an object, with corresponding properties, but in my case the object literal appears as an argument within a top-level (source-file-level) function call

I don't really see why that changes anything. You might easily have some very large object literals forming the bulk of your file, for example.


Automate - Fixed service quota

https://bugzilla.redhat.com/show_bug.cgi?id=1336925

@tinaafitz
Please Review

Since Consolidated quota is used for services, infrastructure and cloud VMs, we had to modify a  previous vendor specific change to determine the vendor differently for services and VM provisioning.  Additionally, generic services should bypass quota checking.
@mkanoor Please review.

@mkanoor
Yes, prov_type is the vendor.

Working on spec tests for service quota.

@mkanoor
Please Review

👍


Initial Implementation

I checked with the maintainers of Smithy4s regarding publishing the generated trait files, and they said it should work fine. There's a note in the docs stating

We cannot recommend treating Smithy4s-generated code as publishable library-material. Should you decide to do so, please exercise caution.

but they told me that's really talking about services and associated datatypes, which change frequently, and not traits, which largely stay the same over time.
I'm going to merge this quickly to get an initial release out for our internal PoC, but I would appreciate a closer review of the SchemaVisitorTraceableValue code if anyone is interested in doing so.

Thanks to @kubukoz and @baccata for feedback in the Smithy4s Discord!


Fix GCC errors on Trusty

Workaround:
// Hack to prevent the stdlibc++ error with clang:
//  /usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/cstdio:120:11: error:
//      no member named 'gets' in the global namespace
//  using ::gets;
#include <initializer_list>  // force libstdc++ to include its config
#undef _GLIBCXX_HAVE_GETS    // correct broken config

Will not fix. Targetting GCC 4.9 instead because an external library depends on it.


Chart.Line  Additional format for the "controlling time interval" prop.   Аllows to set the beginning and end of a range

for prop "controlling time interval using object" of Chart.Line  widget  by default the end date of the interval is the current time (Date.now())
Аdded a new format for string type values like "startTimeStamp;endTimeStamp" where startTimeStamp and endTimeStamp are the start and end timestamps of the interval. This can be used to implement zooming and scrolling of the displayed chart data.

Thanks for the PR!
Did you test it?
I will do a code review in the next few days

//Did you test it?
sure

Please resolve the conflicts.

what is the next step?


Functional Component Transformation (Part 1)

Description
Further transformation associated to #259
Current progress

[ ] /components – 20/32 (+4)
[ ] /jupyter – 0/1 – low priority
[ ] /utils – 4/5 – webserver-utils low priority
[ ] /views – 14/19 (+4)
[x] App.jsx
[x] main.jsx

Checklist

[x] The documentation reflects the changes.
[ ] I have manually tested the application to make sure the changes don’t cause any downstream issues, which includes making sure ./orchest status --ext is not reporting failures when Orchest is running.
[ ] In case I changed code in the orchest-sdk, I updated its version according to SemVer in its _version.py and updated the version compability table in its README.md

[ ] In case I changed one of the services’ models.py I have performed the appropriate database migrations.

⚠ DO NOT MERGE: FilePicker has not been tested

DO NOT MERGE –  ConfigureJupyterLabView is untested ⚠
Having troubles running locally after changing branches

⚠ DO NOT MERGE - There is a crucial bug with unsavedChanges that needs to be addressed as a priority

LGTM! Well done on the transform to functional components.


Suspecting an error in calculateCacheOffset

Hi there!
I was a little confused when reading the source code, not sure if I misunderstood it or there is a missing part in the code. As the code shown below, is the variable b missing a + constraints.cacheOrigin? Otherwise, to my understanding, the cacheOrigin distance will be calculated twice, which make the b bigger than expect.
https://github.com/flutter/flutter/blob/a5da3a2217eb73f000aee6043e50b33dfaf0268f/packages/flutter/lib/src/rendering/sliver.dart#L1491-L1497

Hi @yangyuanxx
The goal of this Github is to improve flutter reporting bugs or feature requests. So if you encounter behavior that you believe could be a Flutter framework bug, please provide a completed and minimal reproducible code sample that represents the problem. Also, please provide the output of flutter doctor -v as well.
Otherwise, please see flutter's community for resources and ask questions like this. Thanks!


clarify CHANGELOG for polygonToCellsExperimental

Clarifies the CHANGELOG entry for polygonToCellsExperimental, since this is not yet implementing polygonToCells

coverage: 98.782%. remained the same
when pulling cea192ac94e2888d190c21803987a2cbcf8847da on isaacbrodsky:changelog-4.2.0-polyfill
into 33681ff65bc4bb4a3addf8dbef5c6b070f811b27 on uber:master.

This is definitely going to cause a merge conflict in my third cli PR


Catch exception in db_sync.get(JobV2, job_id) in submit_workflow

We did observe an error like
sqlalchemy.exc.OperationalError: (psycopg.OperationalError) consuming input failed: terminating connection due to administrator command
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
[SQL: SELECT jobv2.id AS jobv2_id, jobv2.project_id AS jobv2_project_id, jobv2.workflow_id AS jobv2_workflow_id, jobv2.dataset_id AS jobv2_dataset_id, jobv2.user_email AS jobv2_user_email, jobv2.slurm_account AS jobv2_slurm_account, jobv2.dataset_dump AS jobv2_dataset_dump, jobv2.workflow_dump AS jobv2_workflow_dump, jobv2.project_dump AS jobv2_project_dump, jobv2.worker_init AS jobv2_worker_init, jobv2.working_dir AS jobv2_working_dir, jobv2.working_dir_user AS jobv2_working_dir_user, jobv2.first_task_index AS jobv2_first_task_index, jobv2.last_task_index AS jobv2_last_task_index, jobv2.start_timestamp AS jobv2_start_timestamp, jobv2.end_timestamp AS jobv2_end_timestamp, jobv2.status AS jobv2_status, jobv2.log AS jobv2_log
FROM jobv2
WHERE jobv2.id = %(pk_1)s::INTEGER]
[parameters: {'pk_1': 33}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

with the traceback pointing at this line
https://github.com/fractal-analytics-platform/fractal-server/blob/060680e35d8fe74311464ab2f280fb11a4cd8d0f/fractal_server/app/runner/v2/__init__.py#L117

Independently on the reason (in this case it was probably due to an unexpected restart of the postgresql service), let's catch this error and call fail_job if appropriate.

Independently on the reason (...), let's catch this error and call fail_job if appropriate.

We actually cannot call fail_job, as it would likely fail for the same reason (not being able to access the database).
The best we can do is probably to raise a more explicit ERROR log, to clarify where this comes from in the fractal-server logs.

Ah interesting. Makes sense, if we can't connect to the database, we also can't set the job state to failed, in the database 😂


Orderby query with continuationtoken fix.

Fixing an issue due to which the order by results when resuming fro…m continuation token might contain duplicates/missing documents

Adding log to print the current distinct query being tested, to identify intermittent failures on CI

/azp run java - cosmos - tests

E2E test failures due to SSL exceptions on netty
2020-05-11T15:00:48.3887657Z 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-
4.1.49.Final.jar:4.1.49.Final]
2020-05-11T15:00:48.3889696Z 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-reactor.netty.http.client.HttpClientConnect - [id: 0x1e838e09, **] The connection observed an error
2020-05-11T15:00:48.5115307Z javax.net.ssl.SSLException: SSLEngine closed already
2020-05-11T15:00:48.5117202Z 	at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:848) [netty-handler-4.1.49.Final.jar:4.1.49.Final]


Specify which country code is supported

Hello, please can you specify which country code is supported ?

This has none of the features it says that it has except the Mobile number which always comes up invalid

@ThothTheGoat @r4lix Mobile number must include country code but without + sign

Giving invalid mobile number, what can I do

The phone number is showing as invalid even without the + before the number. For example 55<PHONE_NUMBER>1

it only works on Indian numbers


Bow is still too close 1.19.2

Want to use this mod with Better Combat and otherwise, but considering the bow animation is still in a position blocking the player camera, I can't justify ruining that part of gameplay. Any idea when you'll be backporting the Custom V1 animation from 1.6.4 to proper 1.19.2 versions if at all?

I really don't get why people insist on using 1.19.2. No one is asking for 1.18.0, so why 1.19.2(or 1.19.3 for that matter)? I may create a new 1.19.2 release next time I work on Not Enough Animations(wrong repo btw), but it's certainly not a priority, 1.19.4/1.20.1 is.

sorry to necro, but to respond ur question; since 1.20 is still relatively new, there's still a lot of mods in 1.19.2 that haven't been ported yet, plus all the 1.19.2 modpacks, and so people keep playing on this version.

Sorry, but 1.19.2 won't be getting anything new. Also just noticed that this issue is on the wrong mod/git too.

New branch supports 1.16.5-1.20.4 with the same features.


only one blendshape from multiple mesh instances can apply to a shared resource at a time

Godot version
v3.3.4.stable.official [faf3f883d]
System information
Windows 10, GLES3, GeForce 750ti
Issue description
I've got a bunch of meshes instances which should be able to share the same mesh resource but have their own independent blendshapes the same way they can have independent material overrides or shader parameters as I fully expect given that the blendshapes are stored as a mesh instance variable not a mesh resource property. But instead any changes made to one mesh will effect all others.

Steps to reproduce
run the code, pressing enter changes it from 0 to 1 and back. I encourage you to change which wall is set to wall. And check out the values of the blendshapes in the other walls.
Minimal reproduction project
BlendshapesBug.zip

It's not a bug. The mesh can be shared across multiple mesh instances. In order to "fix" this you should simply make it unique:

I know that blender 4.0 is doing a thing where each instance can have its own uniforms. I think a similar structure would work here. Perhaps the code can be reused.

This is already available in 4.0 with per-instance uniforms. It can't be backported to 3.x though.

can't believe I said blender 4.0 I  meant godot 4.0

@fire I hear you're handling the importing stuff. Whats your take on this? Is it even possible to have one mesh resource apply different blendshapes for each instance? In order for it to render would it have to make mesh resources for each time there are different blendshapes applied? Either way it is implemented this should be a 4.0 milestone. Mostly because for all instances sharing one mesh resource only one instance can apply them to the shared mesh and it is often difficult to tell which one can.

Does forcing importer make the mesh resources unique as a fix sound good?
This appeared to be a rendering feature, so the most local solution is to make the importer duplicate the resource.

@fire that sounds like a good quick fix for 3.0 but for 4.0 I think it shouldn't have to compensate by having the blendshapes with the mesh resource from the getgo or even do the mesh details resource thing like I was talking about but that only in the instance that one mesh resource can handle multiple blendshape values. In my game I've got lots of objects who I want to share both mesh and blendshapes so your quick fix doesn't help my scenario. Although I'm probably in a small minority.

@starry-abyss doing so would make it impossible for me to share mesh resources on a bunch of mesh instances which all have the same blendshapes.

I think this should be closed as duplicate of #10402. Or is this a different issue?
Also, how does this issue relate with the proposal godotengine/godot-proposals#2465

@lyuma these are different but probably related bugs. I'm not experiencing any visual glitches like that. In mine its that I've got multiple mesh instances that share a mesh resource and only one of the instance's blend shape values are applied the rest are ignored.

Sharing as in for optimization or sharing as in they are the same visually? I'm not sure how it works in Godot, but I guess it's possible to make it shared in .pck, while have it duplicated in RAM.
Then if you want sharing even in RAM, for rendering optimization (#53244), the temporary solution of just duplicating could be undone as part of the next PR. I don't see how one prevents the other. I understand that the temporary solution is probably not an improvement for your project because of #53244, but there are other users with less blend shapes at the same mesh at once, for whom it's an improvement.

I think this is fixed in Godot 4, I can not reproduce the issue anymore.
Properly because in Godot 4 the BlendShape blend_weights are no longer part of the Mesh Surface but now part of the MeshInstance in the RenderingServer mesh storage so each instance has its own weights.

Closing based on the previous comment


Editor FPS are halved (or more) on AMD cards when set to "Update changes"

Godot 3.0.6
Godot 3.1 alpha4
Windows 7 64 bits
AMD Radeon HD 6670
This is an AMD variant of what I believe is another driver doing things pretty badly when an app does not always render every frame at 60 fps. The twist here, is that it happens even if the editor would have to render every frame anyways. This causes extremely noticeable lag, basically half the FPS it should have, or less.
Reproduction steps (on my current machine):

Create an empty scene, add a DirectionalLight and a MeshInstance
Put a PlaneMesh on the MeshInstance with 100x100 subdivisions
Set the editor to update always using the top-right update spinner
Look at the plane so that it fits in the screen, and pan in circles with the mouse to force the editor to redraw every frame: notice it runs smoothly at 60 fps.
Now, set the editor to "update changes" (which is the default), and redo the same as 4): notice the editor became laggy as if it went down to 30 fps or less. This does not go away after a while, the editor stays at this very low framerate no matter what.

And as usual, this then affects everything (since the editor can do everything), from 3D to script edition.
I even tried to put particles in the scene to force 60 fps, or put TIME in the shader editor, to no avail. It just doesn't want to go above 30.
I know there is a good chance that this is partially dependent on drivers, but when I see that Godot is able to render at 60 fps flawlessly on the other mode... there must be room for improvement in the way Godot actually decides to redraw, especially in cases where it actually HAS to, like shown in the steps above.

This is on purpose to avoid high amounts of CPU usage from the editor.

I know this very well at this point... but how is that related to forcing something like 30 fps or less? Is that also really intented, especially since it doesn't repro on another hardware? Running that slowly when things actually happen in the editor can't be right.
There is some bonus though: since this is Windows 7, I tried to switch off Aero using another theme: it fixed the issue. Why don't things just work these days xD


[maverick] Fix path of RootStore in model template

Also fix the path to the newly-created model that's added to the RootStore
Please verify the following:

[x] yarn ci:test jest tests pass with new tests, if relevant
[x] README.md has been updated with your changes, if relevant

Describe your PR
Creating a new model store on maverick uses an incorrect path for the RootStore leading to an error. This PR fixes the path the the RootStore and also the path to the newly-created model

Thanks @edjw!


[#6084] reword the statement about profiler.sampling.rate in faq.md

Reword the statement about profiler.sampling.rate in faq.md.
Pinpoint release binary has changed the profiler.sampling.rate to 1, but it is still 20 in the document, it might be better to change the doc to a more general statement.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #6087 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #6087      +/-   ##
==========================================
+ Coverage   40.77%   40.78%   +<.01%
==========================================
  Files        3024     3024
  Lines       90599    90599
  Branches    11817    11817
==========================================
+ Hits        36943    36949       +6
+ Misses      50491    50485       -6
  Partials     3165     3165

Impacted Files
Coverage Δ

...vercorp/pinpoint/web/vo/ApplicationAgentsList.java
66.66% <0%> (-2.57%)
:arrow_down:

...rc/main/java/com/navercorp/pinpoint/test/Item.java
70% <0%> (ø)
:arrow_up:

...java/com/navercorp/pinpoint/rpc/DefaultFuture.java
74.16% <0%> (+1.66%)
:arrow_up:

...navercorp/pinpoint/rpc/common/SocketStateCode.java
85.36% <0%> (+2.43%)
:arrow_up:

...m/navercorp/pinpoint/rpc/client/ConnectFuture.java
85% <0%> (+5%)
:arrow_up:

...nitor/DefaultDataSourceMonitorRegistryService.java
88.88% <0%> (+11.11%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8420e7...bf306af. Read the comment docs.

@AlphaWang Thank you very much for your fix and contribution!


Training COCO from Scratch

Thanks for the good work!
Just wanted to mention that I have tried the two currently most stared EfficientDet PyTorch repos on Github, and neither reproduce the paper results on COCO, not even close.
They mostly claim they train on custom data and port weights from official TF checkpoints, but fail to train from scratch on COCO.
I tried their implementations with 200+ kinds of hyper-parameter tuning sets & settings - yes 200+ jobs!
Very keen to see your completed training on COCO. Looking forward to that.
Cheers

Yeah, didn't look like anyone else was close to reproducing training from scratch. Bit of a grind because it takes a long time, looks like I'm getting there, finally a decent D0 result in. Getting pretty close...
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.324
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.513
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.342
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.383
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.280
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.426
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.452
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.188
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.532
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668

Looks good.

Do expect considerable code change for the final release of efficientdet that matches the official implementation?
If not, and if it's only a matter of hyper-parameter tuning, I can kick-off some training jobs to see if I can push the numbers any higher?

@factplay1 the important bits are stable, I have a few misc bugs/mods locally that I should push up first. I'll merge the training branch to master later today once I do that and put up some reference training cmd lines.
Biggest missing thing is still a proper eval so that checkpoint selection can be based on a per-epoch mAP instead of just the loss. That change alone might bump my results to match the official. Since checkpoints with highest mAP don't always have the lowest loss.

Fair enough.

Merged...

Thanks for the pointers
Will give it a try & let you know how it goes ^_^

I ran many jobs.
For some reasons I couldn't get distributed training work on my cloud, so I only ran jobs on 1 GPU.
I tried 20+ different sets of hyper-parameters.
Overall seems unstable at the moment, but seems like batch=32, with weight decay is more stable.

I got a loss of 0.58 for now.  Now training new jobs, but does 0.58 sound like a reasonable value of loss to you? (on validation after 300 epochs)

Would you be able to add support for multi gpu training on a same node?

thanks

My reported results were with the command line posted on the README, for D0 (2 x 2080Ti) ./distributed_train.sh 2 /mscoco --model tf_efficientdet_d0 -b 16 --amp --lr .05 --warmup-epochs 5 --sync-bn --opt fusedmomentum --fill-color mean --model-ema ... for D1 it was similar but with batch size per GPU of 10 (2x Titan RTX)  .. make sure you validate with the fill-color and redundant bias flags as I've left the validation script defaulted to the values for the Google weights.
As you can see it's distributed training, so that works fine on a single node. The training script I adapted from another repo of mine and that has been used successfully by a number of people in the cloud with 8xV100 type scenarios. I have Apex AMP installed with C++ extensions, I'd recommend that. PyTorch native DDP might work but I haven't tested it here.
0.58 isn't that far off for D0, I had about .564 as the low point, started overfitting in the mid 200s and ended up around .58 val loss towards the end. The train script here will retain the best checkpoints.
For D1 the lowest validation losses were in the .531 range, starting to increase in the high 200s.

Sounds good,
thanks for the pointers.
I'll continue playing with the training code then

Hi @factplay1 and @rwightman
Training seems to be very unstable at initialization.
Using 8 GPUs, I keep -b 16 and set --lr 0.20. Training proceeds slowly during the first epoch. At the start of the second epoch, when the learning-rate jumps to 4.8e-2, there is a cascade of gradient overflows and training fails.
Increasing --warmup-lr to 0.01 helped. The loss decreased significantly in the first epoch (from 13422 at initialization down to 2.1). Loss is still decreasing after 9 epochs (0.90), and I have not seen another loss scale adjustment since the 1st epoch.
Seems important for the loss to make significant progress on first epoch before the learning rate jumps up to 4.8e-2 on the second epoch.
Also have stability problems using 1 GPU and -b 32 --lr 0.05. The gradient overflows begin about halfway through the first epoch, when the loss equals about 275. Soon after the loss is nan. The fact that this is logically equivalent to 2 GPUs and -b 16 seems to indicate that even that choice of parameters is not stable.

Yup, first few epochs of training are very touch and go. It usually stabilizes late epoch 0 to sometime epoch 1, or just blows up in your face. Based on comments and FAQ added to the TF official impl, seems like it's also a 'feature' of the original :)
I've got a training session going right now looking positive, with all the latest changes I just merged in... I added clipping of the bbox to image bounds (regardless of the infill/model image size) and found that gives worthwhile gains on validation.
I haven't tried 1 vs two GPU, typically always doing 2 with sync-bn. If I turn off sync-bn it blows up but I assumed that's just due to a small batch size for BN.

@rwightman thanks for the updates. Tried your recent commits, no change in single GPU batch-size 32 training:
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 32 --amp --lr .05 --warmup-epochs 5 --sync-bn --opt fusedmomentum --fill-color mean --model-ema
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
NVIDIA APEX installed. AMP on.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.14s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Train: 0 [   0/3665 (  0%)]  Loss: 11164.052734 (11164.0527)  Time: 4.564s,    7.01/s  (4.564s,    7.01/s)  LR: 1.000e-04  Data: 0.750 (0.750)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 128.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 64.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32.0
Train: 0 [  50/3665 (  1%)]  Loss: 11721.471680 (10832.8051)  Time: 0.711s,   44.98/s  (0.811s,   39.47/s)  LR: 1.000e-04  Data: 0.011 (0.025)
Train: 0 [ 100/3665 (  3%)]  Loss: 11636.480469 (10396.9049)  Time: 0.709s,   45.11/s  (0.760s,   42.10/s)  LR: 1.000e-04  Data: 0.011 (0.018)
Train: 0 [ 150/3665 (  4%)]  Loss: 9315.281250 (9972.0763)  Time: 0.709s,   45.15/s  (0.743s,   43.05/s)  LR: 1.000e-04  Data: 0.011 (0.015)
Train: 0 [ 200/3665 (  5%)]  Loss: 11839.604492 (9897.1951)  Time: 0.708s,   45.18/s  (0.735s,   43.53/s)  LR: 1.000e-04  Data: 0.011 (0.014)
Train: 0 [ 250/3665 (  7%)]  Loss: 8650.294922 (9611.9082)  Time: 0.708s,   45.18/s  (0.730s,   43.84/s)  LR: 1.000e-04  Data: 0.011 (0.014)
Train: 0 [ 300/3665 (  8%)]  Loss: 7696.292480 (9503.3802)  Time: 0.708s,   45.19/s  (0.727s,   44.05/s)  LR: 1.000e-04  Data: 0.010 (0.013)
Train: 0 [ 350/3665 ( 10%)]  Loss: 6910.673828 (9249.6038)  Time: 0.709s,   45.16/s  (0.724s,   44.20/s)  LR: 1.000e-04  Data: 0.011 (0.013)
Train: 0 [ 400/3665 ( 11%)]  Loss: 9258.589844 (9079.8404)  Time: 0.709s,   45.11/s  (0.722s,   44.31/s)  LR: 1.000e-04  Data: 0.010 (0.013)
Train: 0 [ 450/3665 ( 12%)]  Loss: 6829.935547 (8937.0713)  Time: 0.709s,   45.15/s  (0.721s,   44.39/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 500/3665 ( 14%)]  Loss: 6926.620117 (8776.3480)  Time: 0.708s,   45.19/s  (0.720s,   44.47/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 550/3665 ( 15%)]  Loss: 6075.458496 (8630.6923)  Time: 0.708s,   45.19/s  (0.719s,   44.52/s)  LR: 1.000e-04  Data: 0.010 (0.012)
Train: 0 [ 600/3665 ( 16%)]  Loss: 7387.142578 (8476.4539)  Time: 0.708s,   45.19/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 650/3665 ( 18%)]  Loss: 10251.372070 (8338.3930)  Time: 0.715s,   44.76/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 700/3665 ( 19%)]  Loss: 5585.245117 (8184.6530)  Time: 0.714s,   44.84/s  (0.719s,   44.50/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16.0
Train: 0 [ 750/3665 ( 20%)]  Loss: 6334.165527 (8036.1240)  Time: 0.829s,   38.60/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 800/3665 ( 22%)]  Loss: 5322.323730 (7876.7954)  Time: 0.710s,   45.08/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 850/3665 ( 23%)]  Loss: 4763.157227 (7727.7256)  Time: 0.712s,   44.93/s  (0.720s,   44.45/s)  LR: 1.000e-04  Data: 0.012 (0.012)
Train: 0 [ 900/3665 ( 25%)]  Loss: 4682.166016 (7556.3113)  Time: 0.712s,   44.97/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [ 950/3665 ( 26%)]  Loss: 3295.284424 (7375.9236)  Time: 0.716s,   44.71/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1000/3665 ( 27%)]  Loss: 3333.829590 (7175.6596)  Time: 0.711s,   45.03/s  (0.719s,   44.49/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1050/3665 ( 29%)]  Loss: 2858.758545 (6976.0234)  Time: 0.759s,   42.13/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8.0
Train: 0 [1100/3665 ( 30%)]  Loss: 1904.628296 (6769.6703)  Time: 0.710s,   45.08/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1150/3665 ( 31%)]  Loss: 1237.608643 (6560.1093)  Time: 0.712s,   44.93/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.0
Train: 0 [1200/3665 ( 33%)]  Loss: 1429.643921 (6354.0225)  Time: 0.714s,   44.82/s  (0.719s,   44.49/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1250/3665 ( 34%)]  Loss: 1154.844360 (6152.2873)  Time: 0.719s,   44.48/s  (0.720s,   44.45/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1300/3665 ( 35%)]  Loss: 863.659058 (5957.4975)  Time: 0.722s,   44.32/s  (0.720s,   44.46/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1350/3665 ( 37%)]  Loss: 1443.282471 (5770.0738)  Time: 0.754s,   42.45/s  (0.720s,   44.44/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1400/3665 ( 38%)]  Loss: 673.052185 (5589.9676)  Time: 0.715s,   44.79/s  (0.720s,   44.45/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1450/3665 ( 40%)]  Loss: 634.547119 (5420.9808)  Time: 0.710s,   45.04/s  (0.720s,   44.47/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1500/3665 ( 41%)]  Loss: 900.202332 (5260.1546)  Time: 0.712s,   44.96/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.008 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.0
Train: 0 [1550/3665 ( 42%)]  Loss: 545.116028 (5106.5106)  Time: 0.711s,   45.02/s  (0.719s,   44.48/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1600/3665 ( 44%)]  Loss: 379.544373 (4960.4261)  Time: 0.714s,   44.84/s  (0.719s,   44.49/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1650/3665 ( 45%)]  Loss: 297.594666 (4820.6806)  Time: 0.714s,   44.84/s  (0.719s,   44.50/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1700/3665 ( 46%)]  Loss: 346.973389 (4687.7122)  Time: 0.717s,   44.65/s  (0.719s,   44.50/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.5
Train: 0 [1750/3665 ( 48%)]  Loss: 306.872864 (4560.6426)  Time: 0.710s,   45.08/s  (0.719s,   44.51/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.25
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.125
Train: 0 [1800/3665 ( 49%)]  Loss: 170.196304 (4440.0571)  Time: 0.715s,   44.75/s  (0.719s,   44.52/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.03125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.015625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0078125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.00390625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.001953125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0009765625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.00048828125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.000244140625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0001220703125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 6.103515625e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.0517578125e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.52587890625e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.62939453125e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.814697265625e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.9073486328125e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.5367431640625e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.76837158203125e-07
Train: 0 [1850/3665 ( 50%)]  Loss: 193.598160 (4324.8272)  Time: 0.704s,   45.47/s  (0.719s,   44.53/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.384185791015625e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1920928955078125e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.960464477539063e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.9802322387695312e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.4901161193847656e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.450580596923828e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.725290298461914e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.862645149230957e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.313225746154785e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.656612873077393e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.3283064365386963e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1641532182693481e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.820766091346741e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.9103830456733704e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.4551915228366852e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.275957614183426e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.637978807091713e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.8189894035458565e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.094947017729282e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.547473508864641e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.2737367544323206e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1368683772161603e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.684341886080802e-14
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.842170943040401e-14
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.4210854715202004e-14
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.105427357601002e-15
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.552713678800501e-15
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.7763568394002505e-15
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8.881784197001252e-16
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.440892098500626e-16
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.220446049250313e-16
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1102230246251565e-16
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.551115123125783e-17
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.7755575615628914e-17
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.3877787807814457e-17
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 6.938893903907228e-18
Train: 0 [1900/3665 ( 52%)]  Loss: 188.368393 (4215.4645)  Time: 0.699s,   45.78/s  (0.718s,   44.54/s)  LR: 1.000e-04  Data: 0.008 (0.011)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.469446951953614e-18
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.734723475976807e-18
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8.673617379884035e-19
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.336808689942018e-19
# [...]

$ python3 collect_env.py
Collecting environment information...
PyTorch version: 1.5.0+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Quadro RTX 6000
Nvidia driver version: 440.33.01
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.5
/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.17.5
[pip3] torch==1.5.0+cu101
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.6.0+cu101
[conda] Could not collect

Same failure when training with batch-size 48 on 1 GPU.
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 48 --amp --lr .075 --warmup-epochs 5 --sync-bn --opt fusedmomentum --fill-color mean --model-ema
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
NVIDIA APEX installed. AMP on.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.25s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Train: 0 [   0/2444 (  0%)]  Loss: 9457.186523 (9457.1865)  Time: 6.289s,    7.63/s  (6.289s,    7.63/s)  LR: 1.000e-04  Data: 0.954 (0.954)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 128.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 64.0
Train: 0 [  50/2444 (  2%)]  Loss: 8168.729004 (10606.5583)  Time: 1.019s,   47.12/s  (1.144s,   41.96/s)  LR: 1.000e-04  Data: 0.015 (0.033)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32.0
Train: 0 [ 100/2444 (  4%)]  Loss: 8857.475586 (10161.6222)  Time: 1.023s,   46.91/s  (1.082s,   44.36/s)  LR: 1.000e-04  Data: 0.015 (0.024)
Train: 0 [ 150/2444 (  6%)]  Loss: 10997.752930 (10049.3619)  Time: 1.018s,   47.13/s  (1.061s,   45.24/s)  LR: 1.000e-04  Data: 0.015 (0.021)
Train: 0 [ 200/2444 (  8%)]  Loss: 8187.724121 (9770.2847)  Time: 1.016s,   47.27/s  (1.051s,   45.68/s)  LR: 1.000e-04  Data: 0.015 (0.019)
Train: 0 [ 250/2444 ( 10%)]  Loss: 8338.733398 (9473.1848)  Time: 1.020s,   47.06/s  (1.046s,   45.87/s)  LR: 1.000e-04  Data: 0.015 (0.018)
Train: 0 [ 300/2444 ( 12%)]  Loss: 6982.625977 (9275.7555)  Time: 1.069s,   44.90/s  (1.048s,   45.78/s)  LR: 1.000e-04  Data: 0.015 (0.018)
Train: 0 [ 350/2444 ( 14%)]  Loss: 8208.707031 (9104.3530)  Time: 1.021s,   47.02/s  (1.045s,   45.95/s)  LR: 1.000e-04  Data: 0.015 (0.017)
Train: 0 [ 400/2444 ( 16%)]  Loss: 7973.209961 (8923.2916)  Time: 1.017s,   47.21/s  (1.041s,   46.10/s)  LR: 1.000e-04  Data: 0.015 (0.017)
Train: 0 [ 450/2444 ( 18%)]  Loss: 6961.071777 (8767.8622)  Time: 1.084s,   44.29/s  (1.042s,   46.08/s)  LR: 1.000e-04  Data: 0.013 (0.017)
Train: 0 [ 500/2444 ( 20%)]  Loss: 8218.721680 (8629.8574)  Time: 1.018s,   47.17/s  (1.040s,   46.16/s)  LR: 1.000e-04  Data: 0.013 (0.016)
Train: 0 [ 550/2444 ( 23%)]  Loss: 6367.537109 (8497.8610)  Time: 1.018s,   47.14/s  (1.038s,   46.25/s)  LR: 1.000e-04  Data: 0.015 (0.016)
Train: 0 [ 600/2444 ( 25%)]  Loss: 7142.958008 (8383.6200)  Time: 1.013s,   47.39/s  (1.036s,   46.33/s)  LR: 1.000e-04  Data: 0.015 (0.016)
Train: 0 [ 650/2444 ( 27%)]  Loss: 6003.675781 (8258.0362)  Time: 1.011s,   47.47/s  (1.036s,   46.33/s)  LR: 1.000e-04  Data: 0.014 (0.016)
Train: 0 [ 700/2444 ( 29%)]  Loss: 6678.241211 (8127.8840)  Time: 1.013s,   47.40/s  (1.034s,   46.40/s)  LR: 1.000e-04  Data: 0.015 (0.016)
Train: 0 [ 750/2444 ( 31%)]  Loss: 6597.071289 (7992.2111)  Time: 1.017s,   47.18/s  (1.033s,   46.47/s)  LR: 1.000e-04  Data: 0.015 (0.016)
Train: 0 [ 800/2444 ( 33%)]  Loss: 5069.318848 (7841.2663)  Time: 1.008s,   47.60/s  (1.032s,   46.52/s)  LR: 1.000e-04  Data: 0.014 (0.016)
Train: 0 [ 850/2444 ( 35%)]  Loss: 4411.801270 (7682.4960)  Time: 1.014s,   47.34/s  (1.031s,   46.58/s)  LR: 1.000e-04  Data: 0.014 (0.016)
Train: 0 [ 900/2444 ( 37%)]  Loss: 6480.168945 (7521.7178)  Time: 1.008s,   47.63/s  (1.029s,   46.63/s)  LR: 1.000e-04  Data: 0.014 (0.016)
Train: 0 [ 950/2444 ( 39%)]  Loss: 3898.652344 (7347.9578)  Time: 1.011s,   47.46/s  (1.029s,   46.67/s)  LR: 1.000e-04  Data: 0.014 (0.016)
Train: 0 [1000/2444 ( 41%)]  Loss: 3619.440430 (7170.4822)  Time: 1.021s,   47.03/s  (1.028s,   46.70/s)  LR: 1.000e-04  Data: 0.015 (0.016)
Train: 0 [1050/2444 ( 43%)]  Loss: 2690.094727 (6984.2766)  Time: 1.021s,   47.00/s  (1.028s,   46.71/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1100/2444 ( 45%)]  Loss: 2661.113037 (6792.9257)  Time: 1.017s,   47.22/s  (1.028s,   46.70/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1150/2444 ( 47%)]  Loss: 2561.356934 (6601.9898)  Time: 1.068s,   44.93/s  (1.029s,   46.63/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1200/2444 ( 49%)]  Loss: 1595.269775 (6410.7129)  Time: 1.017s,   47.22/s  (1.029s,   46.63/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1250/2444 ( 51%)]  Loss: 1373.499756 (6218.5628)  Time: 1.015s,   47.30/s  (1.031s,   46.57/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1300/2444 ( 53%)]  Loss: 896.425110 (6031.5679)  Time: 1.124s,   42.69/s  (1.031s,   46.56/s)  LR: 1.000e-04  Data: 0.013 (0.015)
Train: 0 [1350/2444 ( 55%)]  Loss: 1078.193970 (5849.8788)  Time: 1.013s,   47.41/s  (1.031s,   46.57/s)  LR: 1.000e-04  Data: 0.012 (0.015)
Train: 0 [1400/2444 ( 57%)]  Loss: 772.888916 (5672.3441)  Time: 1.022s,   46.95/s  (1.030s,   46.58/s)  LR: 1.000e-04  Data: 0.012 (0.015)
Train: 0 [1450/2444 ( 59%)]  Loss: 709.549927 (5502.1795)  Time: 1.020s,   47.04/s  (1.030s,   46.60/s)  LR: 1.000e-04  Data: 0.016 (0.015)
Train: 0 [1500/2444 ( 61%)]  Loss: 492.328278 (5339.0694)  Time: 1.022s,   46.98/s  (1.030s,   46.61/s)  LR: 1.000e-04  Data: 0.016 (0.015)
Train: 0 [1550/2444 ( 63%)]  Loss: 401.367859 (5183.4169)  Time: 1.044s,   45.99/s  (1.030s,   46.59/s)  LR: 1.000e-04  Data: 0.012 (0.015)
Train: 0 [1600/2444 ( 65%)]  Loss: 360.365936 (5034.4046)  Time: 1.039s,   46.22/s  (1.031s,   46.56/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1650/2444 ( 68%)]  Loss: 293.433899 (4892.1026)  Time: 1.029s,   46.65/s  (1.031s,   46.55/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Train: 0 [1700/2444 ( 70%)]  Loss: 234.045761 (4755.9505)  Time: 1.047s,   45.86/s  (1.031s,   46.54/s)  LR: 1.000e-04  Data: 0.015 (0.015)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.5
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.25
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0625
Train: 0 [1750/2444 ( 72%)]  Loss: 150.494934 (4626.1071)  Time: 1.037s,   46.29/s  (1.032s,   46.52/s)  LR: 1.000e-04  Data: 0.013 (0.015)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.03125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.015625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0078125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.00390625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.001953125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0009765625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.00048828125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.000244140625
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0001220703125
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 6.103515625e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.0517578125e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.52587890625e-05
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.62939453125e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.814697265625e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.9073486328125e-06
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.5367431640625e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.76837158203125e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.384185791015625e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1920928955078125e-07
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.960464477539063e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.9802322387695312e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.4901161193847656e-08
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.450580596923828e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.725290298461914e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.862645149230957e-09
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.313225746154785e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.656612873077393e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.3283064365386963e-10
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.1641532182693481e-10
Train: 0 [1800/2444 ( 74%)]  Loss: 188.853699 (4502.5365)  Time: 1.079s,   44.50/s  (1.033s,   46.45/s)  LR: 1.000e-04  Data: 0.014 (0.015)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5.820766091346741e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.9103830456733704e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.4551915228366852e-11
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 7.275957614183426e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.637978807091713e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.8189894035458565e-12
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 9.094947017729282e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4.547473508864641e-13
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.2737367544323206e-13

@andravin I wouldn't expect that last batch of changes to change the early stage blow ups. Validation result improvements and (hopefully) some small img/s increases and slight CPU usage decrease.
The one GPU blow up is quite weird though, especially given that it seems more stable with 2 cards and sync-bn. If you train without AMP does it converge or go to inf? Any difference using the torch momentum opt instead of the Apex fused one? --opt momentum

@andravin re your earlier comment, are you saying in the situation above, if you increase the warmup lr the blowup does not happen? That is very non-intuitive if yes.

I have not found any parameters that make the single GPU training work. It would be interesting if you get different results from me with single GPU training.
I have also found that --clip-grad does not help (at least with float16). The gradient clipping happens after the gradient overflow, too little too late! Would be more effective with bfloat, I think. I wonder if smarter loss scaling could do an infinity-norm clipping of the gradient, including over-flowed values?
Anyway, I am trying float32 training with batch size 28 now (cannot fit 32 in 24MB of VRAM). Will try --opt momentum next.

float32 does not work either, at least with batch-size 28. Will change optimizer next.
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 28 --lr .04375 --warmup-epochs 5 --sync-bn --opt fusedmomentum --fill-color mean --model-ema
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
NVIDIA APEX installed. AMP off.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.20s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Train: 0 [   0/4189 (  0%)]  Loss: 10694.789062 (10694.7891)  Time: 2.586s,   10.83/s  (2.586s,   10.83/s)  LR: 1.000e-04  Data: 0.703 (0.703)
Train: 0 [  50/4189 (  1%)]  Loss: 9156.940430 (10655.5285)  Time: 0.738s,   37.95/s  (0.779s,   35.93/s)  LR: 1.000e-04  Data: 0.009 (0.022)
Train: 0 [ 100/4189 (  2%)]  Loss: 7768.429688 (10266.7713)  Time: 0.743s,   37.66/s  (0.759s,   36.89/s)  LR: 1.000e-04  Data: 0.010 (0.016)
Train: 0 [ 150/4189 (  4%)]  Loss: 9209.789062 (9986.9127)  Time: 0.738s,   37.92/s  (0.761s,   36.80/s)  LR: 1.000e-04  Data: 0.008 (0.013)
Train: 0 [ 200/4189 (  5%)]  Loss: 8507.548828 (9786.8591)  Time: 0.740s,   37.85/s  (0.756s,   37.04/s)  LR: 1.000e-04  Data: 0.008 (0.012)
Train: 0 [ 250/4189 (  6%)]  Loss: 8362.959961 (9689.7940)  Time: 0.740s,   37.84/s  (0.753s,   37.18/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 300/4189 (  7%)]  Loss: 8830.989258 (9470.6703)  Time: 0.741s,   37.79/s  (0.751s,   37.28/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 350/4189 (  8%)]  Loss: 6901.359863 (9382.0202)  Time: 0.737s,   37.97/s  (0.750s,   37.35/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 400/4189 ( 10%)]  Loss: 7986.277344 (9167.3039)  Time: 0.741s,   37.79/s  (0.749s,   37.40/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 450/4189 ( 11%)]  Loss: 6355.049805 (9009.6173)  Time: 0.741s,   37.77/s  (0.748s,   37.41/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 500/4189 ( 12%)]  Loss: 6655.864258 (8863.4486)  Time: 0.743s,   37.69/s  (0.748s,   37.41/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [ 550/4189 ( 13%)]  Loss: 8568.301758 (8704.6989)  Time: 0.737s,   37.97/s  (0.748s,   37.42/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 600/4189 ( 14%)]  Loss: 5678.072754 (8578.0917)  Time: 0.744s,   37.65/s  (0.748s,   37.43/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 650/4189 ( 16%)]  Loss: 5743.519531 (8420.9749)  Time: 0.742s,   37.75/s  (0.748s,   37.44/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 700/4189 ( 17%)]  Loss: 4353.489258 (8266.7042)  Time: 0.738s,   37.94/s  (0.747s,   37.47/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 750/4189 ( 18%)]  Loss: 6153.513184 (8112.7629)  Time: 0.738s,   37.96/s  (0.747s,   37.49/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 800/4189 ( 19%)]  Loss: 4661.080078 (7938.6051)  Time: 0.739s,   37.90/s  (0.746s,   37.52/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 850/4189 ( 20%)]  Loss: 4842.231445 (7766.3439)  Time: 0.741s,   37.76/s  (0.746s,   37.54/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 900/4189 ( 21%)]  Loss: 3944.198730 (7595.5714)  Time: 0.739s,   37.89/s  (0.746s,   37.52/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [ 950/4189 ( 23%)]  Loss: 4084.652100 (7408.9787)  Time: 0.742s,   37.75/s  (0.746s,   37.53/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1000/4189 ( 24%)]  Loss: 2729.191406 (7222.8579)  Time: 0.737s,   37.99/s  (0.746s,   37.54/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1050/4189 ( 25%)]  Loss: 2511.953125 (7020.7226)  Time: 0.738s,   37.96/s  (0.746s,   37.55/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1100/4189 ( 26%)]  Loss: 2250.664551 (6810.5114)  Time: 0.739s,   37.91/s  (0.745s,   37.56/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1150/4189 ( 27%)]  Loss: 2001.307495 (6593.8812)  Time: 0.741s,   37.77/s  (0.745s,   37.57/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1200/4189 ( 29%)]  Loss: 1425.511108 (6380.7197)  Time: 0.738s,   37.92/s  (0.745s,   37.59/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1250/4189 ( 30%)]  Loss: 952.159546 (6170.9877)  Time: 0.739s,   37.87/s  (0.745s,   37.60/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1300/4189 ( 31%)]  Loss: 842.244812 (5969.0061)  Time: 0.738s,   37.93/s  (0.745s,   37.61/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1350/4189 ( 32%)]  Loss: 740.108459 (5776.5566)  Time: 0.739s,   37.91/s  (0.744s,   37.62/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1400/4189 ( 33%)]  Loss: 496.832397 (5593.7206)  Time: 0.740s,   37.86/s  (0.744s,   37.62/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1450/4189 ( 35%)]  Loss: 478.221771 (5419.1644)  Time: 0.825s,   33.94/s  (0.744s,   37.63/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [1500/4189 ( 36%)]  Loss: 395.504578 (5253.0891)  Time: 0.734s,   38.16/s  (0.744s,   37.63/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1550/4189 ( 37%)]  Loss: 255.688934 (5094.7929)  Time: 0.737s,   37.98/s  (0.744s,   37.64/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1600/4189 ( 38%)]  Loss: 236.591843 (4944.3645)  Time: 0.762s,   36.73/s  (0.744s,   37.65/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [1650/4189 ( 39%)]  Loss:       nan (   nan)  Time: 0.737s,   37.99/s  (0.744s,   37.65/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [1700/4189 ( 41%)]  Loss:       nan (   nan)  Time: 0.731s,   38.28/s  (0.743s,   37.66/s)  LR: 1.000e-04  Data: 0.010 (0.009)

--opt momentum does not help
BTW, os Pytorch 1.5 OK?
Will try --clip-grad next.
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 28 --lr .04375 --warmup-epochs 5 --sync-bn --opt momentum --fill-color mean --model-ema
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
NVIDIA APEX installed. AMP off.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.22s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Train: 0 [   0/4189 (  0%)]  Loss: 10694.789062 (10694.7891)  Time: 2.642s,   10.60/s  (2.642s,   10.60/s)  LR: 1.000e-04  Data: 0.706 (0.706)
Train: 0 [  50/4189 (  1%)]  Loss: 9156.939453 (10655.5297)  Time: 0.751s,   37.28/s  (0.786s,   35.61/s)  LR: 1.000e-04  Data: 0.009 (0.022)
Train: 0 [ 100/4189 (  2%)]  Loss: 7768.456055 (10266.7780)  Time: 0.750s,   37.31/s  (0.769s,   36.39/s)  LR: 1.000e-04  Data: 0.009 (0.016)
Train: 0 [ 150/4189 (  4%)]  Loss: 9209.865234 (9986.9331)  Time: 0.752s,   37.24/s  (0.772s,   36.25/s)  LR: 1.000e-04  Data: 0.009 (0.013)
Train: 0 [ 200/4189 (  5%)]  Loss: 8507.581055 (9786.8834)  Time: 0.750s,   37.34/s  (0.768s,   36.48/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 250/4189 (  6%)]  Loss: 8362.970703 (9689.8207)  Time: 0.753s,   37.18/s  (0.769s,   36.41/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 300/4189 (  7%)]  Loss: 8830.975586 (9470.6908)  Time: 0.754s,   37.12/s  (0.766s,   36.53/s)  LR: 1.000e-04  Data: 0.008 (0.011)
Train: 0 [ 350/4189 (  8%)]  Loss: 6901.346680 (9382.0348)  Time: 0.755s,   37.10/s  (0.765s,   36.62/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 400/4189 ( 10%)]  Loss: 7986.270508 (9167.3150)  Time: 0.751s,   37.31/s  (0.764s,   36.65/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 450/4189 ( 11%)]  Loss: 6355.053223 (9009.6274)  Time: 0.753s,   37.18/s  (0.763s,   36.72/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 500/4189 ( 12%)]  Loss: 6655.892090 (8863.4607)  Time: 0.754s,   37.12/s  (0.761s,   36.78/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 550/4189 ( 13%)]  Loss: 8568.620117 (8704.7121)  Time: 0.749s,   37.37/s  (0.760s,   36.82/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 600/4189 ( 14%)]  Loss: 5677.624512 (8578.0913)  Time: 0.751s,   37.27/s  (0.760s,   36.86/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 650/4189 ( 16%)]  Loss: 5742.097168 (8420.9048)  Time: 0.749s,   37.37/s  (0.759s,   36.88/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 700/4189 ( 17%)]  Loss: 4351.906738 (8266.4680)  Time: 0.755s,   37.09/s  (0.759s,   36.91/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 750/4189 ( 18%)]  Loss: 6149.479980 (8112.2119)  Time: 0.755s,   37.07/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 800/4189 ( 19%)]  Loss: 4650.781738 (7937.5174)  Time: 0.749s,   37.37/s  (0.758s,   36.95/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 850/4189 ( 20%)]  Loss: 4813.514160 (7764.6124)  Time: 0.762s,   36.75/s  (0.758s,   36.95/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [ 900/4189 ( 21%)]  Loss: 3944.329346 (7593.6507)  Time: 0.751s,   37.31/s  (0.757s,   36.98/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [ 950/4189 ( 23%)]  Loss: 4068.673096 (7406.6929)  Time: 0.746s,   37.51/s  (0.757s,   37.01/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1000/4189 ( 24%)]  Loss: 2714.164062 (7219.8360)  Time: 0.746s,   37.56/s  (0.756s,   37.03/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1050/4189 ( 25%)]  Loss: 2492.862305 (7017.0259)  Time: 0.746s,   37.56/s  (0.756s,   37.05/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1100/4189 ( 26%)]  Loss: 2252.366699 (6806.2050)  Time: 0.747s,   37.51/s  (0.755s,   37.07/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1150/4189 ( 27%)]  Loss: 1993.562134 (6589.2338)  Time: 0.746s,   37.51/s  (0.755s,   37.09/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1200/4189 ( 29%)]  Loss: 1400.473999 (6376.2767)  Time: 0.749s,   37.37/s  (0.755s,   37.10/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1250/4189 ( 30%)]  Loss: 949.793518 (6166.6029)  Time: 0.745s,   37.57/s  (0.754s,   37.12/s)  LR: 1.000e-04  Data: 0.007 (0.009)
Train: 0 [1300/4189 ( 31%)]  Loss: 837.336792 (5964.3858)  Time: 0.745s,   37.57/s  (0.754s,   37.13/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1350/4189 ( 32%)]  Loss: 735.336487 (5771.5022)  Time: 0.746s,   37.51/s  (0.754s,   37.15/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1400/4189 ( 33%)]  Loss: 491.892761 (5588.7883)  Time: 0.745s,   37.58/s  (0.754s,   37.16/s)  LR: 1.000e-04  Data: 0.007 (0.009)
Train: 0 [1450/4189 ( 35%)]  Loss: 692.349121 (5416.1396)  Time: 0.747s,   37.48/s  (0.753s,   37.17/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1500/4189 ( 36%)]  Loss: 427.876862 (5252.4751)  Time: 0.746s,   37.52/s  (0.753s,   37.18/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1550/4189 ( 37%)]  Loss: 326.174408 (5096.8468)  Time: 0.747s,   37.49/s  (0.753s,   37.19/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1600/4189 ( 38%)]  Loss: 288.869049 (4948.7520)  Time: 0.748s,   37.44/s  (0.753s,   37.19/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1650/4189 ( 39%)]  Loss:       nan (   nan)  Time: 0.738s,   37.94/s  (0.753s,   37.20/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [1700/4189 ( 41%)]  Loss:       nan (   nan)  Time: 0.738s,   37.95/s  (0.752s,   37.22/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1750/4189 ( 42%)]  Loss:       nan (   nan)  Time: 0.749s,   37.40/s  (0.752s,   37.23/s)  LR: 1.000e-04  Data: 0.010 (0.009)

I kicked off a sequence of runs, on PyTorch 1.4 (I'm still using mostly 1.4, but did try 1.5 and didn't see anything off, but speeds were higher on some things and lower on others).
I don't have my bigger RTX cards free, but did some Pascal runs. On a machine that has decent 2 x GPU results, a single GPU, batch size 20 blows up in first epoch with AMP. Without AMP also blows up to NAN (batch size 12). fusedmomentum vs momentum no difference. Trying a higher initial warmup lr.
The BN stats computation with sync-bn may have some differences in order of ops as you compute local vs reduce across nodes, but shouldn't be that significant....

-clip-grad 5.0 did not help.
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 28 --lr .04375 --warmup-epochs 5 --sync-bn --opt momentum --fill-color mean --model-ema --clip-grad 5.0
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
NVIDIA APEX installed. AMP off.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.45s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Train: 0 [   0/4189 (  0%)]  Loss: 10694.789062 (10694.7891)  Time: 2.685s,   10.43/s  (2.685s,   10.43/s)  LR: 1.000e-04  Data: 0.770 (0.770)
Train: 0 [  50/4189 (  1%)]  Loss: 9435.104492 (10803.2935)  Time: 0.759s,   36.90/s  (0.803s,   34.88/s)  LR: 1.000e-04  Data: 0.010 (0.024)
Train: 0 [ 100/4189 (  2%)]  Loss: 8195.472656 (10560.4913)  Time: 0.755s,   37.10/s  (0.782s,   35.79/s)  LR: 1.000e-04  Data: 0.009 (0.016)
Train: 0 [ 150/4189 (  4%)]  Loss: 9920.058594 (10389.4783)  Time: 0.749s,   37.37/s  (0.782s,   35.79/s)  LR: 1.000e-04  Data: 0.010 (0.014)
Train: 0 [ 200/4189 (  5%)]  Loss: 9185.623047 (10265.1682)  Time: 0.750s,   37.35/s  (0.775s,   36.14/s)  LR: 1.000e-04  Data: 0.007 (0.013)
Train: 0 [ 250/4189 (  6%)]  Loss: 9099.480469 (10227.2070)  Time: 0.752s,   37.23/s  (0.770s,   36.35/s)  LR: 1.000e-04  Data: 0.007 (0.012)
Train: 0 [ 300/4189 (  7%)]  Loss: 9660.772461 (10044.3535)  Time: 0.747s,   37.50/s  (0.767s,   36.50/s)  LR: 1.000e-04  Data: 0.008 (0.011)
Train: 0 [ 350/4189 (  8%)]  Loss: 7586.988281 (9995.6754)  Time: 0.749s,   37.36/s  (0.765s,   36.60/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 400/4189 ( 10%)]  Loss: 8889.926758 (9807.6702)  Time: 0.748s,   37.43/s  (0.764s,   36.67/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 450/4189 ( 11%)]  Loss: 7196.492676 (9684.6285)  Time: 0.750s,   37.32/s  (0.762s,   36.74/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [ 500/4189 ( 12%)]  Loss: 7689.833008 (9578.1296)  Time: 0.748s,   37.45/s  (0.761s,   36.79/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [ 550/4189 ( 13%)]  Loss: 10145.991211 (9462.4135)  Time: 0.751s,   37.30/s  (0.762s,   36.73/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [ 600/4189 ( 14%)]  Loss: 6885.529785 (9391.0006)  Time: 0.752s,   37.22/s  (0.762s,   36.77/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [ 650/4189 ( 16%)]  Loss: 7243.470215 (9289.7307)  Time: 0.841s,   33.30/s  (0.762s,   36.77/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 700/4189 ( 17%)]  Loss: 5714.679199 (9200.5878)  Time: 0.800s,   35.00/s  (0.763s,   36.70/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [ 750/4189 ( 18%)]  Loss: 8609.844727 (9123.8382)  Time: 0.757s,   36.98/s  (0.763s,   36.71/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 800/4189 ( 19%)]  Loss: 6887.755859 (9029.4304)  Time: 0.748s,   37.45/s  (0.763s,   36.71/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 850/4189 ( 20%)]  Loss: 7817.358398 (8951.6088)  Time: 0.751s,   37.26/s  (0.762s,   36.73/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 900/4189 ( 21%)]  Loss: 6999.421387 (8895.0906)  Time: 0.752s,   37.24/s  (0.762s,   36.76/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [ 950/4189 ( 23%)]  Loss: 8190.110840 (8827.3342)  Time: 0.747s,   37.50/s  (0.761s,   36.78/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1000/4189 ( 24%)]  Loss: 6264.372559 (8778.2617)  Time: 0.754s,   37.11/s  (0.761s,   36.81/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1050/4189 ( 25%)]  Loss: 6981.790039 (8714.6094)  Time: 0.755s,   37.09/s  (0.760s,   36.83/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [1100/4189 ( 26%)]  Loss: 7669.708496 (8653.4753)  Time: 0.752s,   37.25/s  (0.761s,   36.82/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1150/4189 ( 27%)]  Loss: 8662.011719 (8581.4827)  Time: 0.748s,   37.44/s  (0.760s,   36.84/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1200/4189 ( 29%)]  Loss: 7185.583984 (8518.3628)  Time: 0.750s,   37.31/s  (0.760s,   36.86/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [1250/4189 ( 30%)]  Loss: 6512.355469 (8447.3924)  Time: 0.750s,   37.31/s  (0.759s,   36.87/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1300/4189 ( 31%)]  Loss: 6337.582520 (8372.6215)  Time: 0.750s,   37.33/s  (0.759s,   36.89/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1350/4189 ( 32%)]  Loss: 7205.687500 (8290.9812)  Time: 0.773s,   36.22/s  (0.759s,   36.89/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1400/4189 ( 33%)]  Loss: 5292.782227 (8216.6299)  Time: 0.754s,   37.14/s  (0.759s,   36.88/s)  LR: 1.000e-04  Data: 0.010 (0.010)
Train: 0 [1450/4189 ( 35%)]  Loss: 6015.729980 (8134.1086)  Time: 0.749s,   37.38/s  (0.759s,   36.88/s)  LR: 1.000e-04  Data: 0.009 (0.010)
Train: 0 [1500/4189 ( 36%)]  Loss: 5680.745117 (8049.9625)  Time: 0.755s,   37.11/s  (0.759s,   36.89/s)  LR: 1.000e-04  Data: 0.008 (0.010)
Train: 0 [1550/4189 ( 37%)]  Loss: 4541.784180 (7963.7004)  Time: 0.752s,   37.25/s  (0.759s,   36.90/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1600/4189 ( 38%)]  Loss: 4622.519043 (7874.4832)  Time: 0.753s,   37.21/s  (0.759s,   36.91/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1650/4189 ( 39%)]  Loss: 3754.294922 (7789.5516)  Time: 0.749s,   37.41/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [1700/4189 ( 41%)]  Loss: 2931.083984 (7696.7865)  Time: 0.749s,   37.38/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1750/4189 ( 42%)]  Loss: 3760.308594 (7599.0580)  Time: 0.753s,   37.19/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1800/4189 ( 43%)]  Loss: 3560.346436 (7497.5306)  Time: 0.816s,   34.30/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1850/4189 ( 44%)]  Loss: 3574.380615 (7394.8408)  Time: 0.752s,   37.22/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [1900/4189 ( 45%)]  Loss: 2611.806152 (7283.8720)  Time: 0.753s,   37.20/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [1950/4189 ( 47%)]  Loss: 2791.532227 (7176.3134)  Time: 0.748s,   37.43/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2000/4189 ( 48%)]  Loss: 3079.812988 (7063.0923)  Time: 0.762s,   36.76/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [2050/4189 ( 49%)]  Loss: 2153.663086 (6951.3288)  Time: 0.751s,   37.29/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [2100/4189 ( 50%)]  Loss: 2033.303833 (6835.1761)  Time: 0.755s,   37.09/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.007 (0.009)
Train: 0 [2150/4189 ( 51%)]  Loss: 2792.275635 (6717.0771)  Time: 0.757s,   36.99/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2200/4189 ( 53%)]  Loss: 1376.205322 (6600.3953)  Time: 0.751s,   37.26/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2250/4189 ( 54%)]  Loss: 1094.495605 (6483.5654)  Time: 0.750s,   37.32/s  (0.758s,   36.94/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2300/4189 ( 55%)]  Loss: 1522.181396 (6370.3563)  Time: 0.755s,   37.11/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2350/4189 ( 56%)]  Loss: 1296.643433 (6256.2738)  Time: 0.762s,   36.75/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [2400/4189 ( 57%)]  Loss: 1041.329102 (6146.4666)  Time: 0.754s,   37.12/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2450/4189 ( 59%)]  Loss: 793.629028 (6038.5569)  Time: 0.752s,   37.21/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2500/4189 ( 60%)]  Loss: 826.004211 (5934.0528)  Time: 0.753s,   37.17/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2550/4189 ( 61%)]  Loss: 724.959778 (5831.9345)  Time: 0.752s,   37.22/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2600/4189 ( 62%)]  Loss: 627.059082 (5732.6448)  Time: 0.750s,   37.34/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2650/4189 ( 63%)]  Loss: 499.749115 (5636.0050)  Time: 0.754s,   37.14/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2700/4189 ( 64%)]  Loss: 667.808044 (5542.5843)  Time: 0.749s,   37.38/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2750/4189 ( 66%)]  Loss: 446.831268 (5450.5447)  Time: 0.752s,   37.22/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2800/4189 ( 67%)]  Loss: 405.352020 (5361.3729)  Time: 0.753s,   37.21/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.008 (0.009)
Train: 0 [2850/4189 ( 68%)]  Loss: 358.656006 (5274.4084)  Time: 0.757s,   36.98/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2900/4189 ( 69%)]  Loss: 245.768585 (5189.8677)  Time: 0.757s,   37.01/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [2950/4189 ( 70%)]  Loss: 377.268890 (5107.6028)  Time: 0.753s,   37.20/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [3000/4189 ( 72%)]  Loss: 341.023041 (5027.4170)  Time: 0.776s,   36.08/s  (0.758s,   36.93/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [3050/4189 ( 73%)]  Loss: 280.014557 (4949.3327)  Time: 0.768s,   36.45/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.009 (0.009)
Train: 0 [3100/4189 ( 74%)]  Loss:       nan (   nan)  Time: 0.749s,   37.37/s  (0.759s,   36.91/s)  LR: 1.000e-04  Data: 0.010 (0.009)
Train: 0 [3150/4189 ( 75%)]  Loss:       nan (   nan)  Time: 0.760s,   36.86/s  (0.758s,   36.92/s)  LR: 1.000e-04  Data: 0.010 (0.009)

Can't find a value of --warmup-lr that helps.  After that worked, realized the indent screw up, and then the original works even better, very fast convergence now. Sigh.
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Train: 0 [   0/3665 (  0%)]  Loss: 18.417702 (18.4177)  Time: 4.403s,    7.27/s  (4.403s,    7.27/s)  LR: 1.000e-04  Data: 0.577 (0.577)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Train: 0 [  50/3665 (  1%)]  Loss: 10.774902 (15.8552)  Time: 0.642s,   49.88/s  (0.715s,   44.74/s)  LR: 1.000e-04  Data: 0.009 (0.021)
Train: 0 [ 100/3665 (  3%)]  Loss:  8.739943 (12.3027)  Time: 0.644s,   49.65/s  (0.680s,   47.04/s)  LR: 1.000e-04  Data: 0.009 (0.015)
Train: 0 [ 150/3665 (  4%)]  Loss:  6.261424 (10.4365)  Time: 0.639s,   50.08/s  (0.677s,   47.25/s)  LR: 1.000e-04  Data: 0.009 (0.013)
Train: 0 [ 200/3665 (  5%)]  Loss:  6.263574 (9.3401)  Time: 0.639s,   50.08/s  (0.668s,   47.88/s)  LR: 1.000e-04  Data: 0.009 (0.012)

Still doesn't explain the whole sync bn vs not, but whatever, this certainly trumps all of that nonsense.

@rwightman so glad you found this! Now the learning curve makes a lot more sense. Loss at initialization is much lower now (18.4 instead of 10000) and loss improvements are much smoother during the first epoch. Also, it doesn't explode. :-)
-b 32 --amp looking good so far, will let it run ..
$ python3 train.py /fast-data/coco --model tf_efficientdet_d0 -b 32 --amp --lr .05 --warmup-epochs 5 --sync-bn --opt fusedmomentum --fill-color mean --model-ema
Training with a single process on 1 GPU.
Model tf_efficientdet_d0 created, param count: 3877763
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
NVIDIA APEX installed. AMP on.
Scheduled epochs: 310
loading annotations into memory...
Done (t=12.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.40s)
creating index...
index created!
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0
Train: 0 [   0/3665 (  0%)]  Loss: 18.428669 (18.4287)  Time: 4.708s,    6.80/s  (4.708s,    6.80/s)  LR: 1.000e-04  Data: 0.792 (0.792)
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Train: 0 [  50/3665 (  1%)]  Loss: 10.777158 (15.8640)  Time: 0.743s,   43.08/s  (0.806s,   39.71/s)  LR: 1.000e-04  Data: 0.010 (0.025)
Train: 0 [ 100/3665 (  3%)]  Loss:  8.729959 (12.3059)  Time: 0.721s,   44.41/s  (0.780s,   41.01/s)  LR: 1.000e-04  Data: 0.010 (0.018)
Train: 0 [ 150/3665 (  4%)]  Loss:  6.261559 (10.4385)  Time: 0.711s,   44.99/s  (0.761s,   42.06/s)  LR: 1.000e-04  Data: 0.010 (0.015)
Train: 0 [ 200/3665 (  5%)]  Loss:  6.259273 (9.3415)  Time: 0.714s,   44.81/s  (0.750s,   42.65/s)  LR: 1.000e-04  Data: 0.010 (0.014)
Train: 0 [ 250/3665 (  7%)]  Loss:  5.404747 (8.5312)  Time: 0.715s,   44.73/s  (0.744s,   43.01/s)  LR: 1.000e-04  Data: 0.010 (0.013)
Train: 0 [ 300/3665 (  8%)]  Loss:  4.727291 (7.9684)  Time: 0.722s,   44.30/s  (0.740s,   43.21/s)  LR: 1.000e-04  Data: 0.011 (0.013)
Train: 0 [ 350/3665 ( 10%)]  Loss:  4.560043 (7.5026)  Time: 0.713s,   44.90/s  (0.738s,   43.38/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 400/3665 ( 11%)]  Loss:  4.613047 (7.1410)  Time: 0.714s,   44.84/s  (0.735s,   43.54/s)  LR: 1.000e-04  Data: 0.010 (0.012)
Train: 0 [ 450/3665 ( 12%)]  Loss:  4.222405 (6.8473)  Time: 0.718s,   44.57/s  (0.733s,   43.65/s)  LR: 1.000e-04  Data: 0.010 (0.012)
Train: 0 [ 500/3665 ( 14%)]  Loss:  4.202367 (6.5917)  Time: 0.714s,   44.80/s  (0.731s,   43.75/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 550/3665 ( 15%)]  Loss:  3.898229 (6.3723)  Time: 0.715s,   44.73/s  (0.730s,   43.83/s)  LR: 1.000e-04  Data: 0.009 (0.012)
Train: 0 [ 600/3665 ( 16%)]  Loss:  4.208528 (6.1801)  Time: 0.713s,   44.87/s  (0.730s,   43.85/s)  LR: 1.000e-04  Data: 0.011 (0.012)
Train: 0 [ 650/3665 ( 18%)]  Loss:  4.744430 (6.0126)  Time: 0.717s,   44.61/s  (0.729s,   43.87/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [ 700/3665 ( 19%)]  Loss:  3.719118 (5.8596)  Time: 0.738s,   43.37/s  (0.731s,   43.80/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [ 750/3665 ( 20%)]  Loss:  3.919322 (5.7241)  Time: 0.711s,   45.00/s  (0.730s,   43.84/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [ 800/3665 ( 22%)]  Loss:  3.681147 (5.6010)  Time: 0.777s,   41.17/s  (0.729s,   43.89/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [ 850/3665 ( 23%)]  Loss:  3.666631 (5.4934)  Time: 0.715s,   44.74/s  (0.729s,   43.90/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [ 900/3665 ( 25%)]  Loss:  3.702901 (5.3919)  Time: 0.717s,   44.61/s  (0.729s,   43.91/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [ 950/3665 ( 26%)]  Loss:  3.398074 (5.2998)  Time: 0.724s,   44.21/s  (0.728s,   43.94/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1000/3665 ( 27%)]  Loss:  3.640444 (5.2108)  Time: 0.712s,   44.92/s  (0.728s,   43.96/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1050/3665 ( 29%)]  Loss:  3.574617 (5.1312)  Time: 0.723s,   44.24/s  (0.727s,   43.99/s)  LR: 1.000e-04  Data: 0.012 (0.011)
Train: 0 [1100/3665 ( 30%)]  Loss:  3.214110 (5.0556)  Time: 0.765s,   41.84/s  (0.728s,   43.97/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1150/3665 ( 31%)]  Loss:  3.062830 (4.9840)  Time: 0.713s,   44.89/s  (0.728s,   43.98/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1200/3665 ( 33%)]  Loss:  3.348084 (4.9169)  Time: 0.709s,   45.12/s  (0.728s,   43.98/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1250/3665 ( 34%)]  Loss:  3.365378 (4.8532)  Time: 0.719s,   44.49/s  (0.728s,   43.99/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1300/3665 ( 35%)]  Loss:  3.195876 (4.7942)  Time: 0.715s,   44.75/s  (0.727s,   44.01/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1350/3665 ( 37%)]  Loss:  4.110237 (4.7372)  Time: 0.716s,   44.70/s  (0.727s,   44.03/s)  LR: 1.000e-04  Data: 0.010 (0.011)
Train: 0 [1400/3665 ( 38%)]  Loss:  3.214676 (4.6833)  Time: 0.716s,   44.69/s  (0.726s,   44.05/s)  LR: 1.000e-04  Data: 0.011 (0.011)
Train: 0 [1450/3665 ( 40%)]  Loss:  3.114803 (4.6350)  Time: 0.713s,   44.91/s  (0.727s,   44.02/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1500/3665 ( 41%)]  Loss:  3.470348 (4.5878)  Time: 0.715s,   44.77/s  (0.726s,   44.05/s)  LR: 1.000e-04  Data: 0.009 (0.011)
Train: 0 [1550/3665 ( 42%)]  Loss:  2.949427 (4.5416)  Time: 0.721s,   44.40/s  (0.726s,   44.07/s)  LR: 1.000e-04  Data: 0.011 (0.011)

Hm, now that the first epoch is completing, I am getting a new error in evaluation. This must be a change from one of the earlier commits today.
Train: 0 [3664/3665 (100%)]  Loss:  3.457180 (3.6070)  Time: 3.425s,    5.25/s  (0.722s,   24.92/s)  LR: 1.000e-04  Data: 0.174 (0.011)
/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.
Test: [   0/154]  Time: 1.064 (1.064)  Loss:  2.4959 (2.4959)
Test: [  50/154]  Time: 0.412 (0.428)  Loss:  2.6239 (2.5469)
Test: [ 100/154]  Time: 0.413 (0.421)  Loss:  2.3167 (2.5493)
Test: [ 150/154]  Time: 0.410 (0.428)  Loss:  2.5362 (2.5629)
Test: [ 154/154]  Time: 1.433 (0.434)  Loss:  2.5216 (2.5620)
Loading and preparing results...
DONE (t=3.12s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=41.79s).
Accumulating evaluation results...
DONE (t=15.55s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.002
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.002
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.005
Test (EMA): [   0/154]  Time: 1.070 (1.070)  Loss:  2.5276 (2.5276)
Test (EMA): [  50/154]  Time: 0.378 (0.391)  Loss:  2.6372 (2.5649)
Test (EMA): [ 100/154]  Time: 0.376 (0.384)  Loss:  2.3414 (2.5664)
Test (EMA): [ 150/154]  Time: 0.372 (0.382)  Loss:  2.5519 (2.5790)
Test (EMA): [ 154/154]  Time: 0.376 (0.381)  Loss:  2.5492 (2.5783)
Traceback (most recent call last):
  File "train.py", line 608, in <module>
    main()
  File "train.py", line 448, in main
    lr_scheduler.step(epoch + 1, eval_metrics[eval_metric])
KeyError: 'map'

A simple error that only occurs when --ema is set. Created a new issue here: https://github.com/rwightman/efficientdet-pytorch/issues/18


Get-TwitterUser: Change to auth user only endpoint

Use https://api.twitter.com/2/users/me instead of https://api.twitter.com/2/users/by/username/{0} for Get-TwitterUser.

Closed by 985423d9ac5c9cf59840023a289ec35a7ac07b89