WebsiteAgent is fail sometimes and don't have enough logs

There's an agent who fails every now and then, and I checked logs whereas didn't get enough useful info to troubleshoot:
Message                                         When
Extracting html at .mainMagContent dl dt a: []  5d ago
Extracting html at .mainMagContent dl dt a: []  5d ago
Fetching http://weekly.caixin.com/2016/cw697/  5d ago

And everytime I re-emit the incoming event, it works again. So the only reason I can guess now is the website this agent fetches is down every now and then...but not sure.
So,
A. Is there a way to troubleshoot in detail what cause this problem?
B. If it's external reason to make Website Agent didn't get 200 OK when it runs by self, Any way to make this agent run again later automatically?
This is the scenarios if needed and the agent being talked here is '财新周刊 #2 抓取目录'. Thanks!

Hey @waldens.  I'm not sure.  I'd expect to see an error in the Agent's logs if the site couldn't be reached.  You could also look at the Rails logs (production.log, if you're running in production), but I would expect any error to be in the WebsiteAgent logs.

Thanks @cantino . I've looked at the [production.log and log/*/current](http://waldens.github.io/productionlog and current file.zip) but got no abnormal message associated. I fixed the schedule to 5pm and cancel the 'Propagate immediately' just now, in case that the first minute incoming event come in, the site still couldn't be reached until several hours later. Hope it gonna return back to normal...

It does sound like an external issue, but I'm surprised nothing's in the logs.  Do you see any failed delayed_jobs in the jobs interface at /jobs?

No any failed jobs. And just now, have tested out adding a delay agent to wait ＞12h(fairly enough I think) is still not working. New log:
Message 	                                     When
Extracting html at .mainMagContent dl dt a: [] 	~13h ago
Extracting html at .mainMagContent dl dt a: [] 	~13h ago
Fetching http://weekly.caixin.com/2016/cw698/ 	~13h ago

I'm really stuck here, and happy to provide VPS account if needed.

Are you sure that there is content that matches your selectors? Maybe
they're just not matching anything? The other idea I have is that the site
could be serving different content based on the User Agent. You could try
setting it to Firefox's.
On Mon, Apr 4, 2016 at 1:11 AM Long Li<EMAIL_ADDRESS>wrote:

No any failed jobs. And just now, have tested out adding a delay agent to
wait ＞12h(fairly enough I think) is still not working. New log:
Message                                          When
Extracting html at .mainMagContent dl dt a: []  ~13h ago
Extracting html at .mainMagContent dl dt a: []  ~13h ago
Fetching http://weekly.caixin.com/2016/cw698/   ~13h ago
I'm really stuck here, and happy to provide VPS account if needed.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/cantino/huginn/issues/1401#issuecomment-205184451


Any way to pass extra hosts manually?

I love this for handling all my docker containers hosted using unRAID, but I would really like to also have this container manage a cert for my unRAID GUI itself. I am passing a custom .conf file to configure nginx-proxy to achieve the reverse proxy side, but is there any way to pass this container a manual host name as well?
(or If I get the lets encrypt cert manually, will this container keep it renewed?)

I didn't know unRAID.
The certificate will be renewed if you have a docker container with the right variables (LETS_ENCRYPT_XXXX) and that can be access from the internet through the nginx proxy.


Removed comment that causes false positive

I don't think the comment should be removed, as it's very helpful. I think it should be the user's job to ensure that comments are ignored in their wordlists

I do find it odd that the comment is that lower down in the file (unless it relates to the line directly under it - if so, its not clear . I would also suspect it applies to other lines as well).
I would then opt for it to be moved to the start of the file OR put into README.md (or both ;))

@yechielw are you able to update the pull request?

Please could this MR be fixed up?

Closing due to lack of response :(


Advanced copy, copy wrong record

DBEAVER 7.2.5
Function advance copy.
On a table when is sorted (by click on a column) if you use the fuction advance copy the record copied is not the same as selected.

Thanks for the report.
Closed as duplicate of #9252


update KNN, DT and RF docs to match recent commits

KNN

describe weighted average in more detail

DT & RF

correct some doc errors and omissions
update example to show positive variable importance in RF

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/344/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/347/


Option to overwrite existing prolog

Thank you very much for providing this plugin and continuing to update it! I've only done limited testing, but it appears that saving a changed DITA topic with an existing prolog appends the revised element with the current date. This means that each day we save the topic, the list of revised elements grows. I know we could simply delete the older revised elements, but is there an option to simply replace the existing revised element with the current date?

Hi @mhowes82,
First, thank you for the feedback.
I register an internal issue for this improvement and we will add this in the next version of the add-on.

Has work on the issue above already begun? After working with the tool for a few months now, we think it would also be useful, or more useful than addressing the issue above, to be able to set the number of revised elements to keep in the prolog. For example, we've decided to only keep the two most recent revised elements to prevent the list from growing too long. Would this be a possible change or perhaps a change in place of the issue above? Thanks again for a great plugin!

I added your proposal on our internal issue. We will add an option to control the maximum number of  revised elements that will be kept in the document.


Types not appearing after setting signature group

http://prntscr.com/ops5ky

:grey_exclamation: Duplicate Issue
Thanks for your contribution. There is already an open Issue about this topic. #831 (it will be fixed in the next release)
Feel free to participate in the discussion and share your thoughts in the comments.
:x: Issue closed

Auto-generated response


Indiviual comment deletion

Deletion of individual comments have been added.
Figured it might be necessary rather than deleting single comments using the bulk command.

Awesome!


How to clear LeapMap

There seems no clear method in LeapMap

Hi,
Yeah it's not currently implemented. I will add it when I get a chance.


Eliminate envkey from get_data_dir?

What is the use case for envkey in get_data_dir?
https://github.com/hammerlab/datacache/blob/master/datacache/common.py#L29
It doesn't seem to be used anywhere in the repo, nor in pyensembl, for instance.

I used once to configure which local files to use for IEDB sources but I don't think it's been used since and doesn't actually make too much sense here. I'm happy to remove it.

Now getting used for defining pyensembl data directories.


 I guess it's a slippery slope since so many people stuff env with all sorts of secrets nowadays.

Ended up doing https://github.com/kaihendry/count/commit/d7cd336e06797d6f3bd6402adb86d942e7c1d250

Hmm I think that sort of thing is out of Up's territory, I can't think of many apps where you'd output env variables to a customer or anything like that, definitely white-list instead of black-listing if you have to


Bug: pipette offset problem

The problem
When both pipette try to pick-up a tip from the same rack there is an offset on the pipette not used for calibration;
this offset prevents the pipette from pick-up correctly.
How to reproduce
Load and execute the protocol attached below.
It has been written for an OT-2 loaded with:

left pipette: p300_multi_gen2
right pipette: p300_single_gen2
a 200ul tiprack on slot 8

Just recalibrated deck, pipette offset and tip length.
See the image attached taken using the protocol below: the 8-channel pipette has still offset...

TestTwoPipettesAndTipracks_same_tip.zip
Originally posted by @agostinofacotti in https://github.com/Opentrons/opentrons/issues/7793#issuecomment-839088313

Here a manual way to adjust the offset: inverting by hand the sign of the right pipette x-offset
https://github.com/Opentrons/opentrons/issues/7793#issuecomment-851498862

Still working on parsing through all the data, but I tried to reproduce this last week with a full factory reset and calibration from the ground up, and I was unable to come to a conclusive response. There was a slight difference between the single and multi going to the same tip rack, but not enough of a difference to interfere with tip pickup, nor was it outside our expected margins of tip variance + operator eyeball error

Hi @mcous, thanks for your response 👍
I think the easiest way to see some difference is when you use two P300 multi pipettes on left and right side... In that way you can see 8 tips in a row instead of only one, and it is easy to find the "median" position.
=> see the test protocol attached.
Of course, as you said, tips have dimensional variance; for that when I do the pipette calibration I pay very much attention to use the same tip orientated in the same way (writing a line on the tip, see attached image): I think that calibrating without this is less useful since you're including tip variance in pipette calibration...
Paired-pipette-offset-verify.zip


Possible to use with node ogr2ogr?

Apologies if this is more of a StackOverflow question, but I'm trying to run ogr2ogr to push user-uploaded files to a Postgres database. The ogr2ogr module requires a native install of GDAL.
I've added your Lambda layer to my function but not sure if I need to import or call it. Thoughts?

Wow, sorry I never responded to this @stdmn .
You can use the layer, but you will need to include the ogr2ogr executable in your own deployment.

Also asked in #80 and responded here: https://github.com/developmentseed/geolambda/issues/80#issuecomment-553551347


Fix E notation parsing in BigDecimal

Fixes #9547
These changes fix the various classes of problems i could find by fuzzing the passing of E notation in BigDecimal

Apologies for the long delay on this... I will review if still relevant and fix up PR if needed (else close)

I can confirm this (#9547) is still an issue , and this fix is still relevant.
My RFC for changing the parsing: #9580 #9581 is also thus still relevant but guess it maybe superseeded by #10641 .. anyway will move that discussion there.
Thank you @beta-ziliani for all the work on Crystal (and thanks @straight-shoota for the original review)!
Let me know if there is anything else you need with this PR.


merge: dev to master

:tada: This PR is included in version 0.97.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


RC #191 - Bibliography components

This pull request makes the following changes to the bibliography related components:

Fixes a bug in the useCitationStyles hook with memoization
Pulls the BibliographyForm out into its own component
Creates stories and documentation for the BibliographyForm, BibliographySearchInput, and StyleSelector components

Might be good to document dependencies, like if we have to run a zbib server to support this, if I recall correctly?

Might be good to document dependencies, like if we have to run a zbib server to support this, if I recall correctly?

Done. It looks like the translation server is only required for looking up items via ISBN, DOI, etc, so only the BibliographyList and BibliographySearchInput components require it.


Add Process > Build an Outline

It's been a while, but here's a new section for Prose for Programmers. Would appreciate review if anyone has time. :pencil:

Looks good to me -- happy to see a new section!

@lgierth Thanks for looking it over!


front-generator: TS declarations are not generated for some files

Details from DS required

When writing custom generator, I'd like to reuse some logic and types from existing react-typescript generator. For now I only have custom implementation for app and entity-management templates, so here are the currently missing typedefs:

'@haulmont/jmix-front-generator/lib/generators/react-typescript/app/options'
'@haulmont/jmix-front-generator/lib/building-blocks/stages/writing/pieces/sdk/sdk'
'@haulmont/jmix-front-generator/lib/generators/react-typescript/entity-browser/template-model'
'@haulmont/jmix-front-generator/lib/generators/react-typescript/entity-browser/template-model'
'@haulmont/jmix-front-generator/lib/generators/react-typescript/entity-editor/answers'
'@haulmont/jmix-front-generator/lib/generators/react-typescript/entity-editor/template-model'

But general idea is to have ability to reuse any exported types or functions from existing template if this is the intended use case scenario for writing custom generators.
Also I noticed an error with importing directly from root
node_modules/@haulmont/jmix-front-generator/lib/building-blocks/stages/answers/pieces/defaultAnswers.d.ts:1:40 - error TS2307: Cannot find module 'src/common/studio/studio-model' or its corresponding type declarations.

1 import { StudioTemplateProperty } from "src/common/studio/studio-model";
                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Support Scrapy 2.6.2

The implementation is more complex than I would like due to get_proxyauth being a public method that subclasses may override.
This change does introduce the restriction that get_proxyauth must return a header value of HTTP basic access authentication. As far as I know that is the only authentication method that Zyte Smart Proxy Manager allows.

Scrapy 1.4.0 introduced support for the proxy meta, so I needed to increase the minimum Scrapy version from 1.0.0 to 1.4.0.
Mind that the existing implementation already relies on this feature of Scrapy, so I believe in practice Scrapy 1.4 was already a requirement, but until now we did not have a test that would fail with an earlier version of Scrapy.


[feature] Omni (Sidero) Terraform Provider

Problem Description
As with everything else, it would be great to declaratively configure Omni with Terraform!
Solution
Because there's already a Talos provider, it seems you'd put them both under a common Sidero provider with combined sidero_talos_ and sidero_omni_ resources.

Tracked in https://github.com/siderolabs/omni/issues/53


dart2js: sourcemaps/source_mapping_operators_test fails

sourcemaps/source_mapping_operators_test fails after https://github.com/dart-lang/sdk/commit/d2c403704c5cd13779a00eee22894158a50fdc59
I think the generated code changed because the CL allows a better type to be inferred for .length in args.length > 0.
I don't know how to fix the expected output. Perhaps there should be a README with instructions that work when copy&paste-ed.
Missing code points for function(main) in 'tests/compiler/dart2js/sourcemaps/operators_test_file.dart' in config 'ssa':
  CodePoint[kind=Call (PropertyAccess),js=t2.$gt(),dart=null,location=null]
  CodePoint[kind=Return,js=return t2.$gt(),dart=null,location=null]
Using the new source information system.
Inlining disabled

the test is marked as passing now, I believe this bug is stale.


Update tutorial

Update tutorial to match the code:

need to always send Inputs::None
add more links to other sections
replace Client with Client

Fixed with https://github.com/cBournhonesque/lightyear/pull/49


need more info about this project

is this still in progressing？

Hi, yes, but not as fast as I want to.
Currently I'm working on ICE implementation, which is quite complex to do right. The main problem with WebRTC is connectivity and I want this part to be as good as possible.


Focus is at incorrect location/space on COVID Alert menu with TalkBack on

On Android with TalkBack on, "Close button" focus is lower than at the closing arrow on top for COVID Alert menu.
Android 8.1.0 on Samsung Galaxy
Android 10 on OnePlus 7

Samsung Galaxy J3 Prime
Android version 7.0
Same issue as mentioned above.

@qanitah and @sarasuri This issue has been grouped with focus related issues for the shield expand and collapse. https://github.com/cds-snc/covid-shield-mobile/issues/new


Separator lines of rows of table view disappear/fades when zooming out

🐛 Bug Report
Separator lines of rows of table view disappear/fades when zooming out
🤔 Expected Behavior
It should not fade/disappear
😯 Current Behavior

💁 Possible Solution

🔦 Context
Affecting the UI in ACC Web UI
💻 Code Sample

🌍 Your Environment

Software
Version(s)

react-spectrum

Browser

Operating System

🧢 Your Company/Team
ACC
🕷 Tracking Issue (optional)

This seems to be in a similar vein as https://github.com/adobe/react-spectrum/issues/3017 but TableView doesn't use sub pixel values for its row borders. Unsure if there is anything we can do here, will have to experiment.


depth dense map for test set

could you provide depth map for testing， not for training

Dense depths are only used for training. We do not use depth map for testing, thus we do not generate depth map on test set.


Snowflake trigger not working: choosing warehouse

Expected Behavior
When I use the Snowflake jdbc trigger, I want to be able to choose the virtual warehouse used in that session.
Either via the Kestra parameter "warehouse", or by firing several SQL commands, "USE WAREHOUSE WH; SELECT 1;"
neither is working. only workaround is to give the authenticated user a default warehouse.
(the actual jdbc tasks also seem to work fine, at least for the Query I could set a warehouse and it got used)
Actual Behaviour
Error when using the Kestra parameter "warehouse":
net.snowflake.client.jdbc.SnowflakeSQLException: No active warehouse selected in the current session. Select an active warehouse with the ‘use warehouse’ command.

Error when firing 2 SQL commands in the same session:
net.snowflake.client.jdbc.SnowflakeSQLException: Actual statement count 2 did not match the desired statement count 1.

multi-statement support from Snowflake's SQL-api: https://docs.snowflake.com/en/developer-guide/sql-api/submitting-multiple-statements#specifying-multiple-sql-statements-in-the-request
Steps To Reproduce

using the code from https://kestra.io/plugins/plugin-jdbc-snowflake/triggers/io.kestra.plugin.jdbc.snowflake.trigger
setting additionally a warehouse
changing the sql to "SELECT WAREHOUSE my_warehouse; SELECT * FROM demo_db.public.customers;"

Environment Information

Kestra Version: 0.18.3
Plugin version: ?
Operating System (OS / Docker / Kubernetes): kubernetes
Java Version (If not docker):

Example flow
No response

@MarthaScheffler ping ;) this one is fixed !


Pasting the link the viewer

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)
What did you learn from the changes you have made?
Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?
Were there any challenges? If so, what issue(s) did you face? How did you overcome it?
How were these changes tested?
A reference to a related issue in your repository (if applicable)
Checklist

[ ] I can confirm that my changes are working as intended

I cannot see your homework.

I see it now, You choose all the tables that Thomas chosen in the example logical model and you did not define relationship between them either 1 to 1 or 1 to many.

Do I just add on to this request, adding homework2

I see it now, You choose all the tables that Thomas chosen in the example logical model and you did not define relationship between them either 1 to 1 or 1 to many.

they are all one to many and I added vendor booth and booth assignment as my two.
Your task: choose two tables and create a logical data model.

Homework#3 is done

For HW3 Your Aggregate inquiry is wrong, you don't need to make a nested query after join. Others good

HW4 marked.

Adding hwk 5

HW5 Marked, Passed.

HW6 missing.

Sorry I was on trip. here is hw6

HW 6 Marked Passed.

Assignment Missing.

Sorry, I had a personal emergency which cause the delay.

Assignment marked.


[Design Discussion] TIMESTAMP and TIME ZONE behavior

Hello everyone, I'm creating an issue to track the design discussion of the desired behavior for a date time type that contains time zone information. I conducted a survey of Postgres, MySQL, Trino and SQL Server and noticed that there is no standardized behavior across different SQL systems: https://docs.google.com/document/d/1Dvz7NJ2eliIFm6O1Ra4Azs_WFWR98kHOyDCznIwaXow/edit#
We can essentially choose one of three options:

standardize datetime behavior against an existing system
pick-and-choose behaviors from systems that we think makes the most sense
come up with our own desired behaviors

I believe (1) is likely the best option, or we risk backing ourselves up into inconsistent corners. That option will also allow us to move fastest when implementing additional date time types.
The suggestion on the google doc is to follow Postgres' lead. Please share your thoughts here and on the document!
c.f. https://github.com/apache/pinot/issues/9575 https://github.com/apache/pinot/issues/8045

CC @nizarhejazi @siddharthteotia

Thanks for starting this.
@walterddr @agavra  - Do we also want to use this opportunity to get some consensus on what is our end state w.r.t everything about date/datetime/timestamp related support as first class types.
Example - @vvivekiyer also created issue regarding INTERVAL / TIMESPAN support. Are we considering this as part of discussion ?
I am not saying fold everything into one but may be useful to have discussion at the same time with all the necessary context to see what are the missing ones, what we want to add support for, what we don't want to etc
https://github.com/apache/pinot/issues/9577

@siddharthteotia @vvivekiyer Yes, we want to use this opportunity to discuss the overall handling of all date/time related types. Once we all agree on the high level direction (the convention to follow), the remaining tasks are just implementing each of them.

@siddharthteotia @vvivekiyer Yes, we want to use this opportunity to discuss the overall handling of all date/time related types. Once we all agree on the high level direction (the convention to follow), the remaining tasks are just implementing each of them.

Perfect. Thanks
cc @vvivekiyer

As usual, following the Postgres design looks like a good idea

+1000 on moving towards postgres.


[rtl/pwrmgr] Fix non-existent signals

In lines 280 and 281 of hw/ip/pwrmgr/rtl/pwrmgr.sv there are references to hw2reg.escalate_reset_status,
which don't exist in struct pwrmgr_hw2reg_t in hw/ip/pwrmgr/rtl/pwrmgr_reg_pkg.sv.
This affects the standalone model only, since the top_earlgrey model uses hw/top_earlgrey/ip/pwrmgr/rtl/autogen/pwrmgr_reg_pkg.sv, which fixes the problem.
Perhaps this is a similar problem as in hw/ip/clkmgr, where we need to use lowrisc:systems:clkmgr rather than lowrisc:ip:clkmgr, since the standalone ip model is also incomplete.

@matutem sorry this part is a bit confusing.
if you look here, you'll see that depending on the fileset flag, we pick different register files.
This is kind of unique to the top specific modules.  I actually regret having the model versions under hw/ip now, but for all your DV purposes, could you set the flag to point to the top version?

If the hw/ip/pwrmgr/rtl/pwrmgr.sv is already broken for fileset flag = IP, is there any reason at all to keep it in hw/ip area? We could just put it in hw/top_earlgrey area instead, and get rid of the fileset flag switching. Clearly it's not meant to be used for a different chip implementation (or meant to work standalone).

yep that would be my preference.

I'm downgrading this to P3 as this belongs in the same class of issues where templated modules need to be moved from their legacy location in the hierarchy to ip_autogen. This will remove a significant amount of the "example" confusion.

duplicated by #6519


data_extract task never finishes

Describe the bug
I ran generate_dataset_mapping task and after I have been trying to run extract_dataset task, but the extract dataset task always freezes after processing the last object in the mapping and the only way to get out of it is to abort the process

.
Reproduction steps

cci task run generate_dataset_mapping --org prod
cci task run extract_dataset --org prod

Your CumulusCI and Python versions
CumulusCI version: 3.63.0 (C:\Users\ajani.local\bin\cci)
Python version: 3.9.0 (C:\Users\ajani.local\pipx\venvs\cumulusci\Scripts\python.exe)
Operating System
W10
Windows environment
No response
CumulusCI installation method
No response
Error Gist
No response
Additional information
No response

nevermind, it was just huge amount of data. I will have to find a way to limit that. Maybe through sql filters.


Blank screen on Intel Mac 14.4.1 (23E224)

When opening I'm getting this blank screen, followed all the new installation instructions. Let me know if you need anymore information.

Thank you for reporting this issue, it helps the community a lot.
I managed to identify this on Linux Debian. I believe it is the same case (even for me working on macOS).
Fixed in version 3.0.1, if you persist you can open it again
Thanks!


Use 'users' package

It's mentioned in Spock page in 'works great with' section, shouldn't it be used here?

Yes, that's a great idea! Funblog is older than users (and has not been updated since), that why it does not use it yet. If you or anyone would like to give it a shot: go ahead :-) Otherwise I will look into it when I some time for that...

Hm, I could take a stab as I was about to learn how to use Spock with some kind of authentication.
However, I am beaten by cabal issues :)

You could try using https://github.com/commercialhaskell/stack to skip aroud those issues ;)

Well, I did start with stack. I see that blaze-bootstrap is not on stackage for example.

You can add dependencies that are not on stackage to stack

I think it's not there for a reason, there is an upper bound on blaze-html dependency...sorry, it's just I am not proficient at all in those things:)

Ok, finally built it. Had to fiddle a bit with extra-deps in stack to use my locally built dependencies (as they had some upper bound problems).

Not sure if I'm on the correct way, but I've run into some problems, so I'm looking for help. I've defined a helper:
runUsers action =
  runQuery $ \conn ->
    let b = Users.Persistent (flip runSqlConn conn)
    in action b

Then I'm using it to register user, and it works fine. However, when trying to handle login:
    runUsers $ \b -> Users.authUser b (lr_user loginReq) (Users.PasswordPlain $ lr_password loginReq) 100

I receive sqlite3 returned ErrorError while attempting to perform step, and I nailed it to the fact that authUser uses withAuthUser where actionFn (in this case createToken) runs runPersistent again (so it's run inside another runPersistent).
Is this because of what I chose for runPersistent field in user backend?

What's the error? Are you running two queries in parallel? Can you maybe share all your code?

Hi, my WIP changes are here: https://github.com/BartAdv/funblog/commit/8dc4c91093669289c58782a4cef22968ab6e1186

Hm, currently the tests for persistent run with sqlite, too, w/o a problem. (See https://github.com/agrafix/users/blob/767ebed500a6268826f8550c5e03b5b8223be82d/users-persistent/test/Spec.hs#L18 ). I'd love to look in to it in detail, but it will have to wait until the weekend.

Is this still WIP? The repo by @BartAdv is now missing.

I haven't investigated it further, I've forgot that repo had some stuff from me and I've removed it as I thought it's just a clean fork

I'm currently using Spock together with Users. Maybe this will be useful.
My state looks like this:
data AppState = AppState { userBackend :: Persistent
                         }

And I'm using this helper function:
runUser :: (Persistent -> IO a) -> AppAction ctx foo a
runUser action = do
  state <- getState
  liftIO $ action (userBackend state)

And I use it like this:
-- Create a user
  post ("users") $ do -- Create user
    user <- jsonBody'
    res <- runUser (\b -> createUser (b :: Persistent) (user :: User String))
    case res of
      Right id -> do
        setStatus status200
        text $ pack . show . fromSqlKey $ (id :: UserId Persistent)
      Left err -> do
        setStatus status500
        text . pack . show $ err

I don't know if that is the best way though. The functions in Users takes the backend as their first argument so using runUser is a bit awkward. You have to use a lambda all the time.

I think a good way to fix this would be to switch over to a monadic user interface holding the backend inside a reader.

As in something similar to what Persistent does?
What about simply moving the backend parameter to the end?

Yes, like persistent. Moving the parameter to the end will make the interface akward if you run multiple actions in runUser?


Fix bugs with deactivating a recently renamed stream.

Here's the original report:
https://chat.zulip.org/#narrow/stream/9-issues/subject/Red.20Error.20Message.20appeared.20when.20deleting.20a.20stream.20(Minor)/near/555251
What I can reproduce is that if you do the following:

Rename a stream in "Manage streams"
Go to "manage organization" -> "Delete streams" and delete the renamed stream

Then we throw an exception in the development environment with "Stream deactivation canceled due to non-matching fields." as the error, coming from this code path:
        if ($("#deactivation_stream_modal .stream_name").text() !== $(".active_stream_row").find('.stream_name').text()) {
            blueslip.error("Stream deactivation canceled due to non-matching fields.");
            ui_report.message(i18n.t("Deactivation encountered an error. Please reload and try again."),
               $("#home-error"), 'alert-error');
        }

The actual bug here is that we should be using stream IDs to refer to streams, not their names.  I think if we fixed that, this whole conditional could go away.
I think blueslip.error may be more fatal in development than production (i.e. it may stop execution), which is why I didn't see the ui_report.message here.
The secondary bug is that the ui_report call here shouldn't be using the #home-error element (see the chat.zulip.org thread for how this looks); it should be displaying in an element in the settings form.

@zulipbot claim

@timabbott I am unable to reproduce the error by doing the steps in https://github.com/zulip/zulip/issues/8979#issue-311753343. But I am getting this error when you go to 'organisation settings > delete streams' more than once.

Can you confirm the fact that did you open http://localhost:9991/#organization/streams-list-admin at any point before renaming the stream and then opened it again?

One solution can be to remove event handler using off before attaching any event handler.

Yeah, I think we either do that or we configure the event handlers once globally (rather than every time we render that page).  We do each in some places.


inline link in heading render bad

@mojombo @rtomayko @lsegal
### [rails_panel](https://github.com/dejan/rails_panel)

convert to html

rails_panel\">rails_panel\n

Hi @seaify!
Which version of Redcarpet are you using and with which options please ? I'm not able to reproduce this with 3.3.2:
require 'redcarpet'

render = Redcarpet::Render::HTML.new(with_toc_data: true)
parser = Redcarpet::Markdown.new(render)

markdown = "### [rails_panel](https://github.com/dejan/rails_panel)"
puts parser.render(markdown)

which outputs:

rails_panel

Also please avoid pinging people that are not involved in this project ; they are certainly busy enough. Thanks!

@robin850
finally found out, it's ruby-china's bug, not redcarpet. https://github.com/ruby-china/ruby-china/pull/427.
 class HTMLwithTopic < HTMLwithSyntaxHighlight
      def header(text, header_level)
        l = header_level <= 2 ? 2 : header_level
        %(<h#{l} id="#{text}">#{text}</h#{l}>)
      end
    end

Sorry for pinging them, after create issues and input @, thoes people appear, i thought they also maintain this project.
Thanks for your help!

No problem, thank you for the quick answer! :-)


[Container Instances] ContainerGroups list method result doesn't have instance_view info

Package Name: azure-mgmt-containerinstance
Package Version:  10.0.0
Operating System: Mac OS
Python Version:  3.8

Describe the bug
A clear and concise description of what the bug is.

container_groups of ContainerInstanceManagementClient list method can't collect instance_view info of container filed.

To Reproduce
Steps to reproduce the behavior:

Create Azure Container Instances
Get resource info using ContainerInstanceManagementClient.container_groups.list()
Get resource info using ContainerInstanceManagementClient.container_groups.get()
Compare result and check instance_view info in container field then get result has instance_view info but list result don't

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
Here are screen shots.

list_by_resource_group()

get()

Additional context
I already reported issue here and @Wzb123456789 recommended this issue to transfer here
so if you follow up history of this issue go to the link.

Hi @ImMin5 - Thanks for your report! We'll investigate asap!

according to our investigation, it is the service issue so we transfer it to proper respo: https://github.com/Azure/azure-rest-api-specs/issues/21280. Please wait for the response of service team. Thanks!

/unresolve
Hi, Is this Issue solved?

Hi, @Wzb123456789
I got it..!
Thanks :)


Outdated reference links in carbon console sign-in page

The following has outdated references in the carbon console sign-in page.

Docs
About
User Guide
Sign-in Help

If changing these links cannot be automated let's add this in here[1] to keep track.
https://github.com/wso2/product-is/blob/master/release-guidelines.md

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Add support for day format without leading zero

Format of the day of the month displayed by JTCalendarDayView can now be set without leading zero. Single digit days can be displayed as "1" instead of "01". This is usually the preferred format in the USA.
The date formatter's format is not directly customizable because only the 2 formats "dd" and "d" make sense in this context.
The default behaviour (zero-padded format) has not changed. However one drawback of the current implementation is that it's not possible to toggle between the 2 formats once the calendar has been rendered. This is because, for performance reasons, the date formatter is cached in a static variable instead of being recreated every time. It's unlikely that someone would want to switch between the 2 formats on the fly so it should be good enough for now.

FYI @nmccann

Thank's for your contribution.
I release a new version with what you want but not like you did.
I just exposed the dateFormatter of JTCalendarDayView. So you can change the format of dateFormatter like you want by implementing - (void)calendar:(JTCalendarManager *)calendar prepareDayView:(UIView<JTCalendarDay> *)dayView;.
I want to avoid to put more things in JTCalendarSettings, with time I'm not sure it's a good practice, in the previous version there was font, color, textAlignment...
The settings must be reserved for the logic of the calendar and not something visual.
I will probably code a new version in Swift in the next two months.


Broken weights link

Greetings, I was trying to run test.py, but could not download the model:
❯ python test.py
Loading data.
/home/mscherbina/Documents/github_repos/6DRepNet/dataset/AFLW2000/files.txt
Downloading: "https://cloud.ovgu.de/s/jFd5JAacLdAtZ4N/download/6DRepNet360_300W_LP_AFLW2000.pth" to /home/mscherbina/.cache/torch/hub/checkpoints/6DRepNet360_300W_LP_AFLW2000.pth
Traceback (most recent call last):
  File "/home/mscherbina/Documents/github_repos/6DRepNet360/sixdrepnet360/test.py", line 168, in <module>
    saved_state_dict = load_state_dict_from_url(
  File "/home/mscherbina/anaconda3/lib/python3.9/site-packages/torch/hub.py", line 727, in load_state_dict_from_url
    download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "/home/mscherbina/anaconda3/lib/python3.9/site-packages/torch/hub.py", line 593, in download_url_to_file
    u = urlopen(req)
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/home/mscherbina/anaconda3/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Could you please update the link?

Link updated. https://github.com/thohemp/6DRepNet360/commit/41bad95f059f66cf881825c5f9abd78edf729941

Also, do you mind if I ask when this will be open sourced?

This does not affect the results however, just a sensible thing


Annoying Perl warnings in recent builds.

In recent builds of 1.1.1, there are many warnings output, all:
Use of uninitialized value in pattern match (m//) at ./util/add-depends.pl line 40.

Hmmm, how fresh is your checkout?  I cherry-picked the fix yesterday: 02d3c6aecc646872af1286144ce8af0693a9f4e3

Oops, I see them too.  Darn...

Another question: does this happen only when configured no-shared?

I never use "no-shared".

I found the problem, and it wasn't tied to no-shared.  PR coming up!

See #7635

Addressed by PR. Closing.

You didn't have to close this manually, github would have done that automatically when i finally pushed 0c594cc, because the commit message contains Fixes #7634.  Maybe you would have closed manually anyway, and that's of course fine!  This is just a tip that you don't have to bother if you don't want to.


Fix crash with improper constraint for fuel time

Closes #162 by changing Short.MAX_VALUE to Integer.MAX_VALUE.

Converting to draft - this PR will also end up mixing into furnace serialization to use ints instead of shorts to avoid the source of all this!

...I put the new stuff on the wrong branch in my fork lol, fixed now

By the way, I've tested this PR on a Vanilla client in a QSL server (a LAN server, more specifically); There are no serious issues on there, only visual weirdness on the client-side that shouldn't interfere on anything.


What was the use of import and export command?

There is a command named /hero import and /hero export. I can't seem to find a help with that.  So I also need to know all my commands. If the video finished I'll reach to you so you can put it on the Spigot page.

Hi Clamor, /hero import and /hero export are used for switching between storage types. You use /hero export and it dumps the content of the current database being used into a file, and then you change the storage type you're using, reload, and then /hero import to restore the data.

Alright, thanks @Xemorr


Does Bulk Query stream include all batches?

Does the Bulk Query .stream() include all batches? https://jsforce.github.io/document/#bulk-query
var batch = conn.bulk.load("Account", "insert");
// MULTIPLE BATCHES -> ->
batch.on("response", function(rets) { // fired when batch finished and result retrieved
  for (var i=0; i < rets.length; i++) {
    if (rets[i].success) {
      console.log("#" + (i+1) + " loaded successfully, id = " + rets[i].id);
    } else {
      console.log("#" + (i+1) + " error occurred, message = " + rets[i].errors.join(', '));
    }
  }
);
//
// When input stream becomes available, pipe it to batch stream.
//
// -> -> ARE ALL INCLUDED HERE? -> ->
csvFileIn.pipe(batch.stream());

Also, is there a spec on what events batch.stream() emits?

Also, if you use this style-
conn.bulk.query("SELECT Id, Name, NumberOfEmployees FROM Account")
  .stream().pipe(fs.createWriteStream('./accounts.csv'));

How do you get the .on('progress') event to know when each poll request is sent out? The stream returned by .stream() doesn't seem to expose this.

@tejasmanohar Unfortunately the current bulk query implementation (ver 1.6) doesn't handle multiple batch result (of course it should do).

Hmmm so what happens if multiple results come back in the batch query response? is only the first one used? @stomita

oh snap, that is the case (ref). that should probably be in the docs given that it has the potential to drop data

In what case are multiple batches returned? They can be returned at any time, right? Not specified by the user?


License

Which license is this crate? Cargo.toml says Apache but README.md says MIT. Or is it both such as most Cargo crates?

@jethrogb You bring up a good point. I don't think I am allowed to re-license code without contributors' agreement so I will ask every contributor here.
To the following people, do you agree to license your contributions under the terms of the Apache 2.0 license ?
@dcuddeback
@equalsraf
@petevine
@psych0d0g
@retep998

No objection here.

Either or both license is fine by me.

Either case is fine by me

Either is okay with me.

Thanks a lot for your quick replies @petevine @retep998 @equalsraf and @dcuddeback ! @psych0d0g, what do you think ?

Terribly sorry to bother you with this @psych0d0g but I think I'm legally required to ask for your consent here. Would you be OK with relicensing your contribution as Apache 2.0 ?

@conradkleinespel Try emailing @psych0d0g with the email address attached to one of their commits.

His commits are merely some changes to Cargo.toml that from the looks of it might not be copyrightable.

Especially since all of those lines have since been rewritten anyway, so even if it was copyrightable it no longer applies to the current version.

Sorry for taking so long. @jethrogb and @retep998, you are 100% right. I've clarified licensing so it now clearly says Apache 2.0 everywhere. Thanks all !


chore: Publish after release for correct version

Fixing a small defect from #3022 - currently the stated version on our demo page is one version behind.
In the release workflow, we need to force publish to happen after release, so that the version has been correctly updated before publishing.

Visual-diff flake: d2l-input-date-time-range skeleton wide-hidden-labels-values


Data association with Global Nearest Neighbor Filter not working as expected

I've found that using a global nearest neighbor filter as a data associator in my own code creates an unexpected effect.
Basically, the Global Nearest Neighbor associates the measurement to the first track, rather than the track with the smallest distance.
In reality, this might just be user error, but I wanted to make an issue demonstrating what I'm seeing so that I can be sure this is not a problem with Stone Soup.
See the attached files and script as a demonstration of what I mean (you may have to change the path in the script, so that it points at the pickle files correctly, depending on where you put them).
issueDemonstrationFiles.zip

Thanks for the report @arickGrootveld.
So the issue we've identified is when there is no missed distance set in the DistanceHypothesiser, the default missed distance is then np.inf. This results in the Joint Hypotheses, which sums the combination of individual hypotheses across all tracks, having every sum containing a np.inf in the case that there are less tracks than measurements.
If you use the GNNWith2DAssignment (which we generally recommend to use, as it's much faster) it will raise that the problem is infeasible. The GlobalNearestNeighbour should really also do this, but currently it just returns the first result.
In the interim, adding a large missed distance in the DistanceHypothesier will fix this. In case of using Mahalanobis, then 100 would be fine.
We are thinking a possible fix, would be where the missed distances are all np.inf that setting them to the largest non-missed distance plus epsilon, will ensure that both GlobalNearestNeighbour and GNNWith2DAssignment function correctly in case of having no missed distance defined.

Thank you so much for getting back to me about this @sdhiscocks. I just verified that setting the "missed_distance" parameter in the initialization of the "DistanceHypothesiser" with a large value (I used 100 as you recommended) fixes the issue entirely.
Would you or your group be interested in having an external pull request for this? It might take me a bit to familiarize myself with your code base, but I could at least give it a shot.
Thanks again for helping me with this, and let me know if you guys would be ok with me taking a shot at fixing it.

Would you or your group be interested in having an external pull request for this? It might take me a bit to familiarize myself with your code base, but I could at least give it a shot.

Yes, that would be great. Thanks.


ssh no longer works after post windows update restart

Environment
10.0.19041.0
Windows build number: [run `[Environment]::OSVersion` for powershell, or `ver` for cmd]
Your Distribution version: [On Debian or Ubuntu run `lsb_release -r` in WSL]
Whether the issue is on WSL 2 and/or WSL 1: [run `cat /proc/version` in WSL]

Steps to reproduce
Launch WSL 1 (Ubuntu-20.04), run sudo service ssh start
attempt to log into system either from localhost or from a machine on the network. This worked fine before the reboot required by the most recent .NET install. Though, windows update did also install a few other things like security updates.
I tried turning all 3 parts of windows firewall off, but the behavior didn't change.
$ sudo service ssh start

Starting OpenBSD Secure Shell server sshd                                                                            /etc/ssh/sshd_config line 16: Deprecated option UsePrivilegeSeparation
/etc/ssh/sshd_config line 19: Deprecated option KeyRegenerationInterval
/etc/ssh/sshd_config line 20: Deprecated option ServerKeyBits
/etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication
/etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication
[ OK ]
(base) est@~ $ ssh localhost -vvvvv
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "localhost" port 22
debug2: ssh_connect_direct
debug1: Connecting to localhost [<IP_ADDRESS>] port 22.
debug1: Connection established.
debug1: identity file /home/est/.ssh/id_rsa type 0
debug1: identity file /home/est/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
kex_exchange_identification: Connection closed by remote host
(base) est@~ $ sudo service ssh status

lxcore-logs.zip

Expected behavior
I should be able to ssh into this machine

Actual behavior
SSH doesn't work

10.0.19041.0
Launch WSL 1 (Ubuntu-20.04)

Can't reproduce here on 20180, WSL1, Ubuntu-20.04.

Might be #4898, but I didn't dive enough to confirm (mostly because it worked).
It probably isn't the Windows Update. Or at least, not without some additional unknown external factor contributing to the fail. If it is the Windows Update, there will be a flood of me2s soon enough (but that seems less likely until they arrive).
A couple of things to try:

Set PasswordAuthentication yes in /etc/ssh/sshd_config to see if you can connect without the key exchange.
Install 18.04 and see if it is okay there.

So, password authentication was already enabled and 18.04 behaved the same after I installed it. However, uninstalling two Windows Updates from Jun 21 (computer hadn't been restarted until this morning) and manually shutting down WSL did work. So, things are good now.

uninstalling two Windows Updates from Jun 21

What were the updates you uninstalled.

Sadly, I didn't write down the numbers, but they were the only two windows updates that were installed on Jun 21 (probably when my the window I had instructed windows not to install updates had ended) so whatever would have become available once it was allowed to look for updates. I tried to find references to 'uninstall' in Application events, but nothing shows up. Do you have a way to find out without turning windows update back on? They were not marked as security updates nor were they .net updates, but regular windows updates.

Alright, thanks. Glad you are unblocked.


fix node network deserialization

& test fixtures / mocks for PVE 7.3.3 and 8.0.4
it looks like PVE is not fully adhering to what is specified in their rest api documentation

pve not adhering to their own docs?! shocked... i tell you... shocked /s
great work and thanks for the PR!


47 deployment on ares script

A script single_user_deploy.sh is added to allow deployment of ChronoLog components: ChronoVisor, ChronoKeeper and a Client app on Ares.
It supports both command line argument configuration of all binaries, host files, and job id.
It also supports generating host files automatically from SLURM environment variables.

first of all. I created this script with the mindset that it is just for you and me to be able to test ChronoLog on Ares. now I understand it is more towards the end users. I will remove the hard-coded paths.
as I said previously, I did not expect this script to be used by anyone other than us. that is why I put it in test directory. I will move it to ChronoLog/deploy
first, they are just the default binaries to use when it's run without any command line argument. you can set it to anything you want. second, we do not have a full client executable for now. I will use the targets chronovisor_server, chronokeeper and storyteller_test instead since they complete a basic triangle.
for some reason, sleep() and getpid() need their header files to build. on my system. I don't know how it can work on your side without.

we do not actually need to install the shared library (.so) files in install. we can do it in the deploy script. for example, call ldd on the executable, copy all shared library files to the destination directory, then launch the executables.


Upgrading from TF 2.1 to 2.2 gives 12% slowdown and 23% memory increase

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution: Ubuntu 18.04
TensorFlow version (use command below): 2.1.0, 2.2.0, tf-nightly
Python version: 3.6.9
CUDA/cuDNN version: 10.1
GPU model and memory: 8xTesla V100, 32GB each

Describe the current behavior
I'm running language modeling experiments with ALBERT, and GPU memory is at a premium due to the large batch sizes necessary. Upgrading from TF 2.1.0 to 2.2.0, I experienced OOM errors, so I ran a few benchmarks:

TF nightly (May 8)
TF 2.2
TF 2.1

Iterations/Sec
1.63
1.64
1.86

GPU Memory
26 GB
25 GB
21 GB

That's a combination of 12% speed slowdown, and 23% memory increase. I cannot upgrade until performance is matched. Are there any new experimental options, or changes I should be aware of that caused this massive performance hit? I'm using tf.function, XLA, and AMP.
It seems that fewer and fewer ops are converted to mixed-precision as we progress from TF 2.1->2.2->nightly. Is that related, and how can I restore the original behavior?
Poring over the release notes, the only thing that sticks out is:
tf.constant always creates CPU tensors irrespective of the current device context.

@jarednielsen,
In order to expedite the trouble-shooting process, could you please provide the complete code to reproduce the issue reported here. Thanks!

Sure, replication on this one is a bit complicated since it requires comparing across TF versions and with different config options. Run
pip install tensorflow==2.2.0
pip install<EMAIL_ADDRESS>
Then with the following self-contained training script:
# performance.py
"""
Demonstrates a performance regression from TF 2.1 to 2.2 when using model.call(training=True).
Memory increases from from 11.7GB to 17.6GB and it/s decreases from 7.42 to 6.24.
Use the parameters --batch_size=16 --accum=1, and try with & without --training.
"""

import argparse
import time

import numpy as np
import tensorflow as tf
from tqdm import tqdm
from transformers import AlbertConfig, TFAlbertForPreTraining

def train_batch(input_dict, training: bool):
    with tf.GradientTape() as tape:
        mlm_logits, sop_logits = model(input_dict, training=training)
        loss = tf.reduce_mean(mlm_logits) + tf.reduce_mean(sop_logits)
        scaled_loss = opt.get_scaled_loss(loss)
    scaled_grads = tape.gradient(loss, model.trainable_variables)
    opt.apply_gradients(zip(opt.get_unscaled_gradients(scaled_grads), model.trainable_variables))

def benchmark_lm(num_epochs: int, training: bool):
    start_time = time.perf_counter()
    for i, batch in tqdm(enumerate(dataset.take(num_epochs))):
        train_batch(batch, training)
    tf.print(f"Execution time: {time.perf_counter() - start_time}")

def get_synthetic_mlm_unlabeled_dataset(batch_size: int) -> tf.data.Dataset:
    """ Returns a dataset that includes batching, but not gradient accumulation. """

    def gen(batch_size):
        shape = [batch_size, 512]
        input_ids = tf.constant(np.random.randint(10, size=shape), dtype=tf.int32)
        attention_mask = tf.constant(np.random.randint(2, size=shape), dtype=tf.int32)
        token_type_ids = tf.constant(np.random.randint(2, size=shape), dtype=tf.int32)

        input_dict = {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "token_type_ids": token_type_ids,
        }

        yield input_dict

    dataset = tf.data.Dataset.from_generator(
        gen,
        output_types={
            "input_ids": tf.int32,
            "attention_mask": tf.int32,
            "token_type_ids": tf.int32,
        },
        args=(batch_size,),
    )
    dataset = dataset.repeat()
    return dataset

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--eager", action="store_true")
    parser.add_argument("--batch_size", type=int)
    parser.add_argument("--training", action="store_true")
    args = parser.parse_args()

    eager = args.eager
    batch_size = args.batch_size
    training = args.training

    gpus = tf.config.list_physical_devices("GPU")
    tf.config.experimental.set_visible_devices(gpus[0], "GPU")
    tf.config.experimental.set_memory_growth(gpus[0], True)

    tf.config.optimizer.set_jit(True)  # XLA
    tf.config.experimental_run_functions_eagerly(eager)  # AutoGraph
    tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})  # AMP

    train_batch = tf.function(train_batch)

    config = AlbertConfig.from_pretrained("albert-base-v2")
    model = TFAlbertForPreTraining(config)
    opt = tf.keras.optimizers.Adam(0.01)
    opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, loss_scale="dynamic")

    dataset = get_synthetic_mlm_unlabeled_dataset(batch_size)

    # Cache function
    train_batch(next(iter(dataset)), training)
    benchmark_lm(num_epochs=50, training=training)

Run the following:
python performance.py --batch_size=16
python performance.py --batch_size=16 --training

while monitoring GPU utilization with watch -n 1 nvidia-smi.
Then repeat the above, but with TF 2.1 installed.
The output on my Tesla V100 (32GB RAM) is

Simple Benchmark
2.1 it/s
2.2 it/s
2.1 memory
2.2 memory

training=False
7.13
7.09
11.7GB
11.7GB

training=True
7.12
6.43
11.7GB
17.6GB

Note that TF 2.1 performance is consistent with training=False and training=True, while TF 2.1 drops in speed by 10% and increases memory usage by 50%. The language model being benchmarked uses dropout and layer normalization, but no batch normalization. Thus I suspect that activating dropout is the source of poor performance on TF 2.2.
@amahendrakar Are you able to reproduce the issue with the above code?

I've benchmarked this effect with various model sizes, by changing the number of hidden layers. Happy to share more code if necessary. These are the results:

training=True
2.1 memory
2.2 memory

2 layers
3.2GB
3.2GB

4 layers
5.3GB
5.3GB

6 layers
5.3GB
9.4GB

8 layers
5.3GB
9.4GB

10 layers
9.4GB
9.4GB

12 layers
9.4GB
17.6GB

When training=False, the 2.2 memory results are identical to 2.1. Yet dropout probabilities are set to 0, so the value of training should be irrelevant.
It appears that the amount of memory required increases at discrete intervals, and training=True increases the amount of memory required by TF 2.2 (sometimes doubling it!).
@gowthamkpr Are you able to reproduce the issue?

Memory usage wise, I am surprised that in TF 2.1, training=True consumes the same amount of memory as training=False, in keras dropout layer, training=False means that no dropout is done and thus no dropout mask needs to be preserved for backprop. I would assume training=True will always consume more memory, due to dropout (unless XLA optimize away the dropout mask)
Could you please run the model with only AMP (no XLA) on training=True/False and see how this change memory usage? Thanks.

Also it would be better if you could also try out Keras mixed precision API
https://www.tensorflow.org/guide/keras/mixed_precision

The model is using dropout_prob=0, so there should be no difference between training=True and training=False.
I am using the Keras mixed-precision API, aside from tf.keras.mixed_precision.experimental.set_policy('mixed_float16'), which fails due to dtype mismatches:
tf.config.optimizer.set_experimental_options({"auto_mixed_precision": True})  # AMP
opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, loss_scale="dynamic")
scaled_loss = opt.get_scaled_loss(loss)
scaled_grads = tape.gradient(loss, model.trainable_variables)
opt.apply_gradients(zip(opt.get_unscaled_gradients(scaled_grads)

However, mixed-precision is not the issue because the same memory bug occurs when using full-precision. The memory bug also appears when not using XLA. The memory bug also occurs when not using XLA and not using AMP.
Are you able to run the code above? I tried to make it a minimal reproducible example.

which fails due to dtype mismatches:

CC @reedwm

I also found memorry issue once i upgrade from GPU 2.0 to 2.2. I already reduce half of the data size. But still getting on model.fit. The Nvidia GPU will just disappear in Window.

~\Anaconda3\envs\Transformer_1D-CNN_Feature_Extraction_tf2.2\lib\site-packages\tensorflow\python\profiler\profiler_v2.py in stop(save)
    107     if save:
    108       try:
--> 109         _profiler.export_to_tb()
    110       except Exception:
    111         _profiler = None

MemoryError: bad allocation

@sanjoy The mixed-precision dtype mismatch is a problem, but it's orthogonal to the performance regression. The performance regression occurs in full-precision as well. Have you replicated the issue?

It seems that fewer and fewer ops are converted to mixed-precision as we progress from TF 2.1->2.2->nightly. Is that related, and how can I restore the original behavior?

I can reproduce and also observe AMP is converting less nodes to float16. Assigning to myself.

@reedwm can you also take a look at the regression @jarednielsen mentioned above that seems unrelated to mixed precision?

@reedwm Any insight on the problem or how to sidestep it?

Actually, I believe that this might be an issue of TF2.2 in general, since I am experiencing an effect where a (very simple) transfer learning model works perfectly and fast with or without gpu on TF2.1 and completely fails on TF2.2 both with or without gpu. While it still trains the model, it takes 71s for the first epoch with TF 2.2, where it took 7s with TF2.1. And finally, on TF2.1, it takes one second for inference using the model on 20 previously unseen samples, where this process never finishes with TF2.2.
However, there might be something I missed, but I do find it strange that this just "happens" in a notebook where I did not change a line of code ...
Sorry for being verbose, but not posting the code, but I believe that this is not easily reproduced without the data. However, if there is more I could do to provide additional info, please let me know.

Can you compare with 2.3 now that it's almost near release?

Hi,
I am sorry to say that I am not sure how to install 2.3. If you mean installing tfp-nightly via pip, I did that just now (I am, however, using conda for everything else) an reran the notebook. I definitely see an improvement here, since now the first epoch of training has only a small overhead compared to the other epochs and those are very fast. However, I again had to interrupt the prediction step (which takes only one second using TF2.1 as mentioned above), since it wouldn't finish.
As I write this, I notice that the install command only installed the cpu-version of tensorflow - so I have to check the gpu-version as well concerning the problem with the first epoch.
Anyway, I did install tf-nightly-gpu just now (which gives a TF 2.4, if I see this correctly) and there it complained about the installed scipy version being too new (The Warning message spoke about Tensorflow 2.2.0 needing scipy 1.4.1 - so I wonder whether this might be the problem?). So I downgraded that and I can report that everything worked just fine with this setup (still only CPU, since I need to figure out a way to not have to install the whole CUDA stuff manually - conda does that for me normally).
Once I find the time to approach this more systematically, I will get back. And, before I forget, thanks for getting back to me.
Once I figured out a way

So, I will describe my experiments briefly, but the gist is that with tensorflow==2.3.0rc2 (as well as with the nighly of July 25th, 2020) inference works as it did/does with TF2.1 and the problem is NOT due to the scipy version.
Detailed Experiments
Now on to the detailed experiments, which all start with a fresh miniconda install and the creation of virtual environments for the setups. In all cases, I first do
conda install jupyterlab pandas numpy scipy scikit-learn matplotlib seaborn
First setup (TF-2.2):
Here I install TF as it comes with conda:
conda install tensorflow-gpu
When I train and test my model, I experience both the slow start of training (only when using GPU) as well as not being able to carry out the prediction step on the test data (both with GPU as well as CPU only).
Second setup (TF-2.1):
Here I install TF in a downgraded conda version:
conda install tensorflow-gpu==2.1.0
When I train and test my model, I experience a moderately slow start of training (only when using GPU), but everything else works just fine.
Third setup (TF-2.3rc2 / TF-nightly):
Here I first install TF as usual (to obtain cudnn, cupti, and cudatoolkit), only to remove it and replace it with the pip version. Also, I downgrade scipy from 1.5.0 to 1.4.1 in order not to get a warning from the pip install:

conda install tensorflow-gpu
conda uninstall --force tensorflow tensorflow-base tensorflow-gpu scipy
pip install scipy==1.4.1 tensorflow-gpu==2.3.0rc2

When I train and test my model, I experience a very slow start of training (only when using GPU), but everything else works just fine. The same is true if I replace tensorflow-gpu==2.3.0rc2 with tf-nightly-gpu.
Additional Information on the slow start of training
So, the problem with the slow start of training does NOT happen, if I use the CPU only, but it does happen when using the GPU  (and it seems to have been reported before, where the error message has been reported here). It should be noted that with TF2.1, the error message just reads
Not found: ./bin/ptxas not found Relying on driver to perform ptx compilation. This message will be only logged once.
and then TF more or less just gets on with its business (first epoch takes ~7s, so there is a moderate holdup, but this is normal and also happens in further reduced form when using the CPU only) , while for all later versions there is a long holdup before training starts (first epoch takes between 35s - 71s).
Warning: You might want to refrain from trying next part ...
Now, it seems as if one can get around this problem by reading all of this Stackoverflow discussion and, immediately after installing TF via conda, doing a combination of two answers:
conda install -c conda-forge cudatoolkit-dev
This will update the standard cudatoolkit and provide both nvccas well as ptxas.
However, there are at least two caveats (and one Ooops - see below)

Please note that apparently this package can be installed in one virtual environment only. Since in order to be able to install it in the environment with TF-2.3rc2, I first had to remove the environment with the nightly where I had originally installed it first (maybe uninstalling the Toolkit would have been enough, but I did not want to keep the nightly anyway).

Also, the virtual environment will have its own cudatoolkit in its pkgs directory. However, a lot of the shared library files in the lib64 directory have length zero, so, unsurprisingly, the GPU cannot be used. This can be remedied by copying over everything from the global cudatoolkit lib64 directory, but this stell leaves quite a few zero length files, which might play some part somewhere. For running the training, however, everything seems to be ok.

And now to the really bad part - the Ooops: It seems that the installation of the cudatoolkit-dev messes with your system and not just with miniconda! After installing it, nvidia-detector crashes. While I am not sure how serious this actually is (the system is still running and also works as expected after reboot, at least for now, afaict). So, if you do this, you do it at your own risk!
So, sorry for the long post, but maybe this actually answers @jarednielsen 's original question and also offers a way to sidestep the issue, albeit at a certain cost ...

So, if I understand the comments correctly, TF 2.3 no longer has the slowdown and memory increase, right?
Thank you for the detailed response, it's been very helpful. Do you recall what the scipy warning was?

Hi, in fact, I did not check the memory increase, but there was no more slowdown with TF2.3rc2.
The scipy message was something along the lines that tensorflow requires scipy 1.4.1, while scipy 1.5 is installed and the latter was incompatible (which, in fact, I doubt).
I believe, however, that the cudatoolkit which is installed along with tensorflow, should be upgraded, so the dev-version does not have to be installed in order for tensorflow (or rather the possibleoptimizations) to work properly.

Oh, the scipy message is from pypi/pip. It can safely be ignored

TF 2.3 still has the same speed and memory slowdowns as 2.2. Any update @sanjoy @reedwm?

Also experienced the same issue while upgrading from TF 2.0 to TF 2.3. The slowdown was almost 100% compared to the older version. Is it possible to share some optimization tricks that can help improve model performance? I have also been getting the warning regarding 'batch operation faster than "on_train_batch_end" (not sure if that is related to the slowdown)
example (the actual times vary based on model size, batch size, validation set, etc.
WARNING:tensorflow:Callbacks method 'on_train_batch_end' is slow compared to the batch time (batch time: 0.0811s vs on_train_batch_end' time: 0.1891s). Check your callbacks.'

tensorflow version== '2.5.0-dev20201109'
cuda 11
GPU  RTX 3090
when i run the code over 7-8 hours ,  i would get this error------ "MemoryError: bad allocation"
Traceback (most recent call last): File "train_mspeech.py", line 53, in ms.TrainModel(datapath, epoch = 50, batch_size = 16, save_step = 500) File "D:\ASR_project\asr\SpeechModel251.py", line 187, in TrainModel self.TestModel(self.datapath, str_dataset='train', data_count = 4) File "D:\ASR_project\asr\SpeechModel251.py", line 250, in TestModel pre = self.Predict(data_input, data_input.shape[0] // 8) File "D:\ASR_project\asr\SpeechModel251.py", line 326, in Predict r1 = r[0][0].eval(session=tf.compat.v1.Session()) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 921, in eval return _eval_using_default_session(self, feed_dict, self.graph, session) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 5515, in _eval_using_default_session return session.run(tensors, feed_dict) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 968, in run run_metadata_ptr) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run feed_dict_tensor, options, run_metadata) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run run_metadata) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call return fn(*args) File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 1358, in _run_fn self._extend_graph() File "D:\ASR_project\asr\venv\lib\site-packages\tensorflow\python\client\session.py", line 1398, in _extend_graph tf_session.ExtendSession(self._session) MemoryError: bad allocation

Faced OOM Error upon running the code in this comment with Tensorflow Version 2.5. Please find the Gist.

Hi There,
This is a stale issue. As you are using an older version of tensorflow, we are checking to see if you still need help on this issue. Please test the issue with the latest TensorFlow (TF2.7 and tf-nightly). If the issue still persists with the newer versions of TF, please feel free to open it in keras-team/keras repository by providing details about the issue and a standalone code to reproduce the issue. Thanks!
Please note that Keras development has moved to a separate Keras-team/keras repository to focus entirely on only Keras. Thanks!

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


無校對了的資料，簽名愛提掉

#107

Coverage decreased (-44.02%) to 52.607% when pulling 5ce4b8ba02f44c4a97ca959ea5541a985f605b07 on 無校對了的資料簽名愛提掉 into 3dd647054f9e52e27db80de3a50765cd12a10e12 on master.


Test flakiness on "Check "is_closed_" failed"

Sample failure:
[ RUN      ] SmartCardConnectorApplicationTest.ShutdownWithActiveClient
[DEBUG] Performing services initialization...
[DEBUG] [PC/SC-Lite NaCl port] Initialization...
[DEBUG] [PC/SC-Lite NaCl port] Setting up PC/SC-Lite logging...
[INFO] ../../../../../third_party/pcsc-lite/src/src/debuglog.c:390:DebugLogSetLevel() debug level=debug
[INFO] ../../../../../third_party/pcsc-lite/src/src/debuglog.c:411:DebugLogSetCategory() Debug options: APDU
[DEBUG] [PC/SC-Lite NaCl port] PC/SC-Lite logging was set up.
[DEBUG] [PC/SC-Lite NaCl port] Allocating reader structures...
[DEBUG] [PC/SC-Lite NaCl port] Reader structures allocation finished with the following result: "Command successful.".
[DEBUG] [PC/SC-Lite NaCl port] Performing initial hot plug drivers search...
[DEBUG] [emulated IPC] A blocking in-memory file 1 was created
[DEBUG] [emulated IPC] A blocking in-memory file 2 was created
[DEBUG] [emulated IPC] The in-memory file 1 connected to the in-memory file 2
[DEBUG] [emulated IPC] The in-memory file 2 connected to the in-memory file 1
[DEBUG] [libusb] libusb_init#196(ctx=NULL): called...
[DEBUG] [libusb] libusb_init#196: returning "LIBUSB_SUCCESS" [0x00000000]
[DEBUG] [libusb] libusb_get_device_list#197(ctx=libusb_context<DEFAULT>, list=0xF3AFEFE0): called...
[DEBUG] [libusb] libusb_get_device_list#197: returning 0, *list=0xEF384970([])
[DEBUG] [libusb] libusb_free_device_list#198(list=0xEF384970([]), unref_devices=1): called...
[DEBUG] [libusb] libusb_free_device_list#198: returning
[DEBUG] [emulated IPC] A blocking in-memory file 3 was created
[DEBUG] [emulated IPC] A blocking in-memory file 4 was created
[DEBUG] [emulated IPC] The in-memory file 3 connected to the in-memory file 4
[DEBUG] [emulated IPC] The in-memory file 4 connected to the in-memory file 3
[DEBUG] [emulated IPC] The in-memory file 1 was closed
[INFO] ../../../../../third_party/pcsc-lite/src/src/hotplug_libusb.c:492:HPEstablishUSBNotifications() Driver ifd-ccid.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
[DEBUG] [emulated IPC] The in-memory file 2 was closed
[DEBUG] [emulated IPC] The in-memory file 1 was destroyed
[DEBUG] [emulated IPC] The in-memory file 2 was destroyed
[DEBUG] [PC/SC-Lite NaCl port] Initial hot plug drivers search finished with the following result code: 0.
[DEBUG] [PC/SC-Lite NaCl port] Registering for hot plug events...
[DEBUG] [PC/SC-Lite NaCl port] Registering for hot plug events finished with the following result code: 0.
[DEBUG] [PC/SC-Lite NaCl port] Allocating client structures...
[DEBUG] [PC/SC-Lite NaCl port] Client structures allocation finished with the following result code: 1...
[DEBUG] [PC/SC-Lite NaCl port] Waiting for the readers initialization...
[DEBUG] [PC/SC-Lite NaCl port] Waiting for the readers initialization finished.
[DEBUG] [PC/SC-Lite NaCl port] Starting PC/SC-Lite daemon thread...
[DEBUG] [PC/SC-Lite NaCl port] PC/SC-Lite daemon thread has started.
[DEBUG] [PC/SC-Lite NaCl port] Initialization successfully finished.
[DEBUG] All services are successfully initialized, posting ready message...
[DEBUG] [PC/SC from foo (id 1234)] Created client handler
[DEBUG] [PC/SC-Lite clients manager] Created a new client handler for foo (handler id 1234)
[DEBUG] [PC/SC-Lite NaCl port] [daemon thread] Waiting for the new connected clients...
[DEBUG] [PC/SC from foo (id 1234)] Started processing request SCardEstablishContext(0x02,null,null)...
[INFO] [PC/SC from foo (id 1234)] SCardEstablishContext#199(dwScope=SCARD_SCOPE_SYSTEM, pvReserved1=null, pvReserved2=null): called...
[DEBUG] [emulated IPC] A non-blocking in-memory file 5 was created
[DEBUG] [emulated IPC] A non-blocking in-memory file 6 was created
[DEBUG] [emulated IPC] The in-memory file 5 connected to the in-memory file 6
[DEBUG] [emulated IPC] The in-memory file 6 connected to the in-memory file 5
[DEBUG] [PC/SC-Lite NaCl port] [daemon thread] A new client was connected, starting a handler thread...
[DEBUG] [PC/SC-Lite NaCl port] [daemon thread] Waiting for the new connected clients...
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:341:ContextThread() Authorized PC/SC client
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:345:ContextThread() Thread is started: dwClientID=6, threadContext @0xf412b8b0
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:363:ContextThread() Received command: CMD_VERSION from client 6
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:375:ContextThread() Client is protocol version 4:4
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:397:ContextThread() CMD_VERSION for client 6
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_clnt.c:610:SCardEstablishContextTH() Server is protocol version 4:4
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:363:ContextThread() Received command: ESTABLISH_CONTEXT from client 6
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard.c:210:SCardEstablishContextServer() Establishing Context: 0x776F9744
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:466:ContextThread() ESTABLISH_CONTEXT for client 6
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_clnt.c:3254:SCardAddContext() Allocating new SCONTEXTMAP @0xf4130660
[INFO] [PC/SC from foo (id 1234)] SCardEstablishContext#199: returning 0x00000000 ["Command successful."], hContext=0x776F9744
[DEBUG] [PC/SC from foo (id 1234)] Request SCardEstablishContext finished successfully with the following results: [0x00, 0x776F9744]
[DEBUG] [PC/SC-Lite clients manager] Deleted all 1 client handlers
[DEBUG] [PC/SC-Lite NaCl port] Shutting down the PC/SC-Lite daemon thread...
[DEBUG] [PC/SC-Lite NaCl port] [daemon thread] Shutting down...
[DEBUG] [PC/SC from foo (id 1234)] Performing forced cleanup: canceling all pending blocking requests for left context 0x776F9744
[INFO] ../../../../../third_party/pcsc-lite/src/src/hotplug_libusb.c:501:HPEstablishUSBNotifications() Polling forced every 1 second(s)
[DEBUG] [libusb] libusb_exit#200(ctx=libusb_context<DEFAULT>): called...
[DEBUG] [libusb] libusb_exit#200: returning
[INFO] ../../../../../third_party/pcsc-lite/src/src/hotplug_libusb.c:544:HPEstablishUSBNotifications() Hotplug stopped
[DEBUG] [emulated IPC] The in-memory file 3 was closed
[DEBUG] [emulated IPC] The in-memory file 4 was closed
[INFO] ../../../../../third_party/pcsc-lite/src/src/readerfactory.c:1374:RFCleanupReaders() entering cleaning function
[INFO] ../../../../../third_party/pcsc-lite/src/src/winscard_svc.c:156:ContextsDeinitialize() remaining threads: 1
[DEBUG] [PC/SC-Lite NaCl port] The PC/SC-Lite daemon thread shut down.
[DEBUG] [emulated IPC] The in-memory file 5 was destroyed
[FATAL] Check "is_closed_" failed. File "../src/public/ipc_emulation.cc", line 56, function "~InMemoryFile"

I haven't seen this since a while, and it might've been fixed by our improvements of the PC/SC-Lite shutdown logic.


Add cryptocurrency icons to the project

This PR adds all the cryptocurrency icons into the project instead of retrieving them from an external URL.

Can you revert the changes on package.json and package-lock.json?

@hantuzun Is there any other changes required here? For Issue #10

Thanks @TeenaBatra. Looks good to me. Next time, you can request reviews from me too.


KubeCon NA 2022 - Kiosk and SWAG for Open Feature

Hi Everyone,
as we will have a Kiosk for Open Feature at kubecon it would be useful to know who all will be there so we can have it covered fro the whole conference. If you could add the dates you will be there and your availability we can coordinate.
Also it would be nice to have some SWAG to give away, would anyone be interested in participating here? @beeme1mr had already found a good supplier for Kubecon EU.

We should be good to go for KubeCon.


paasio: Extract common counter code

The "count bytes/operations" portions of the ReadCounter and
WriteCounter were completely analogous, and can be extracted.

Again, the contribution guidelines say that reference solutions don't need to be particularly great code. I balance this against the current situation that they are named "example" which means they're meant to serve as an example... so we should strive to set somewhat good examples, and that means we should at least try to keep the code DRY.

Extracted from #236

I am quite sure that again, git will play it safe and say this will have a merge conflict with #235 . Should be easy to understand since they don't actually modify any same lines (just lines that are very close together), but of course I'm totally ready to handle rebases.


⚠ Bofrost.fr has degraded performance

In 3a0abea, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 1928 ms

Resolved: Bofrost.fr performance has improved in d7bf744 after 6 minutes.


Merge master branch to prTesting1_202011031923 branch

Docs Build status updates of commit d93f2f6:
:clock10: Pending: waiting for processors (6 builds ahead of you)
:warning: Docs Build is busy, currently there are 6 builds ahead of this one, for more information you can view the Build queue graph on the Docs Portal.

Docs Build status updates of commit d93f2f6:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestCon1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Replace (most) warnings with errors

At the moment if Juggle identifies a problem with the query it tries to widen the query and continue. This often results in very long results with the reported error lost at the top of the output.
For example, specifying an unknown type is generalised into a search for java.lang.Object, so a simple typo when looking for methods that return a Thread ends up listing most of the methods in the JDK: $ juggle -r Tread.
Instead of emitting a warning, Juggle should report the unknown class as an error, and perhaps suggest its old "fix" as something the user could try themselves.

Re-enable this test in COVERAGE.md, which is presently reporting thousands of results:
% juggle -j build/libs/testApp.jar 'com.angellane.juggle.testinput.app.App()'
*** Ignoring class com.angellane.juggle.testinput.app.App: java.lang.NoClassDefFoundError: com/angellane/juggle/testinput/lib/Lib
*** Couldn't find type: com.angellane.juggle.testinput.app.App; using class java.lang.Object instead
*** Ignoring class com.angellane.juggle.testinput.app.App: java.lang.NoClassDefFoundError: com/angellane/juggle/testinput/lib/Lib
public Object.<init>()
%


Dotmesh pods not starting on kubeadm on aws

On Centos 7
I0625 21:25:28.933712       1 main.go:369] Starting Dotmesh Operator version release-0.5.4
I0625 21:25:29.234451       1 main.go:471] Labelling unfamiliar node ip-172-20-0-230.us-west-2.compute.internal so we can bind a Dotmesh to it
I0625 21:25:29.253011       1 main.go:471] Labelling unfamiliar node ip-172-20-0-96.us-west-2.compute.internal so we can bind a Dotmesh to it
I0625 21:25:30.270464       1 main.go:1002] Creating pod server-ip-172-20-0-96.us-west-2.compute.internal running quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a on node ip-172-20-0-96.us-west-2.compute.internal
I0625 21:25:30.295779       1 main.go:1002] Creating pod server-ip-172-20-0-230.us-west-2.compute.internal running quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a on node ip-172-20-0-230.us-west-2.compute.internal
I0625 21:25:54.429963       1 main.go:604] Observing pod server-ip-172-20-0-230.us-west-2.compute.internal - status Failed: FAILED (Message: ) (Reason: )
I0625 21:25:54.429997       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:25:54.430029       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [dotmesh-outer]"}
I0625 21:25:54.430044       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:25:54.430059       1 main.go:614] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - container v1.ContainerStatus{Name:"dotmesh-outer", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(0xc4200df1f0)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:"quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a", ImageID:"docker-pullable://quay.io/dotmesh/dotmesh-server@sha256:6b468bdc8310405100ed06ccf06e409a589c34f8dce27e702376f8227a0aa2db", ContainerID:"docker://75176f33f91c824c917b805e319804015d35a438a12d8c62591d4e828e05b62b"}
I0625 21:25:54.430092       1 main.go:685] Deleting pod server-ip-172-20-0-230.us-west-2.compute.internal
I0625 21:25:54.959406       1 main.go:726] Not creating a pod on undotted node ip-172-20-0-230.us-west-2.compute.internal, as the old pod is being cleared up
I0625 21:25:55.959608       1 main.go:604] Observing pod server-ip-172-20-0-96.us-west-2.compute.internal - status Failed: FAILED (Message: ) (Reason: )
I0625 21:25:55.959633       1 main.go:611] Failed pod server-ip-172-20-0-96.us-west-2.compute.internal - condition v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:25:55.959661       1 main.go:611] Failed pod server-ip-172-20-0-96.us-west-2.compute.internal - condition v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [dotmesh-outer]"}
I0625 21:25:55.959673       1 main.go:611] Failed pod server-ip-172-20-0-96.us-west-2.compute.internal - condition v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558730, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:25:55.959684       1 main.go:614] Failed pod server-ip-172-20-0-96.us-west-2.compute.internal - container v1.ContainerStatus{Name:"dotmesh-outer", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(0xc4200dfb90)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:"quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a", ImageID:"docker-pullable://quay.io/dotmesh/dotmesh-server@sha256:6b468bdc8310405100ed06ccf06e409a589c34f8dce27e702376f8227a0aa2db", ContainerID:"docker://e4b2a3eded008e3910d7d7f1a0cb6724d8427d5b6ec6fd3300e37546a6af0f59"}
I0625 21:25:55.959712       1 main.go:685] Deleting pod server-ip-172-20-0-96.us-west-2.compute.internal
I0625 21:25:56.180479       1 main.go:726] Not creating a pod on undotted node ip-172-20-0-96.us-west-2.compute.internal, as the old pod is being cleared up
I0625 21:25:56.180513       1 main.go:1002] Creating pod server-ip-172-20-0-230.us-west-2.compute.internal running quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a on node ip-172-20-0-230.us-west-2.compute.internal
I0625 21:25:57.513183       1 main.go:1002] Creating pod server-ip-172-20-0-96.us-west-2.compute.internal running quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a on node ip-172-20-0-96.us-west-2.compute.internal
I0625 21:26:00.922816       1 main.go:604] Observing pod server-ip-172-20-0-230.us-west-2.compute.internal - status Failed: FAILED (Message: ) (Reason: )
I0625 21:26:00.922841       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558756, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:26:00.922865       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558756, loc:(*time.Location)(0x187b440)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [dotmesh-outer]"}
I0625 21:26:00.922878       1 main.go:611] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - condition v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63665558756, loc:(*time.Location)(0x187b440)}}, Reason:"", Message:""}
I0625 21:26:00.922888       1 main.go:614] Failed pod server-ip-172-20-0-230.us-west-2.compute.internal - container v1.ContainerStatus{Name:"dotmesh-outer", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(0xc420258770)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, Ready:false, RestartCount:0, Image:"quay.io/dotmesh/dotmesh-server:bd09ac301a70bfe2a4669900fa23330844caa77a", ImageID:"docker-pullable://quay.io/dotmesh/dotmesh-server@sha256:6b468bdc8310405100ed06ccf06e409a589c34f8dce27e702376f8227a0aa2db", ContainerID:"docker://1b72ec2ad67625224fade3e3bb9b517caf8ea49ca1783b4c591d323e968ea759"}

Hrm, I think I need to add more logging to find out why these pods are failing, as they're not leaving a useful message. Watch this space...

Please let me know and I will update dotmesh and try.

Hi there! I've built a version with the new logging in, which is available at https://get.dotmesh.io/unstable/465-operator-logging/ - you'll want the YAML files in https://get.dotmesh.io/unstable/465-operator-logging/yaml/dotmesh-k8s-1.8.yaml and so on (just follow the normal upgrade instructions, but use https://get.dotmesh.io/unstable/465-operator-logging/yaml where you'd normally use https://get.dotmesh.io/yaml in the URLs).
It should put a lot more stuff in the logs of the dotmesh-operator deployment - if you can send us what it emits, I can take a look and try to figure out what it's unhappy about!
Thanks!

HI Alaric, Please give us a few days and we will try again. We are busy with a release.

hey @datacabinet any chance you could take a look at this please? If you need help, jump on Slack and ping me and I'll help you out :)

We are extremely sorry for going silent. We are trying to do a release and
struggling with some last minute bugs.
We want to move to k8s asap and would love to get to work with your team.
Regards
On Tue, Jul 3, 2018, 3:13 AM lukemarsden<EMAIL_ADDRESS>wrote:

hey @datacabinet https://github.com/datacabinet any chance you could
take a look at this please? If you need help, jump on Slack and ping me and
I'll help you out :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotmesh-io/dotmesh/issues/465#issuecomment-402092600,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbOF3EOnX4YgkYiB1p8km-Kao6BmeJagks5uC0PigaJpZM4U25E4
.

No worries - looking forward to collaborating. I'll close this issue for now, but please re-open it as soon as you're ready.

I deployed using the new files and I could see some pods active in dotmesh namespace, but one of them(server-ip-172-18-0-163.us-west-2.compute.internal) stayed in ContainerCreating state for ever.
root@ip-172-20-0-160:/app# kubectl get pods -n dotmesh
NAME                                                READY     STATUS              RESTARTS   AGE
dotmesh-dynamic-provisioner-5994f8b745-gvtz9        1/1       Running             0          1d
dotmesh-etcd-cluster-m89wlb22jl                     1/1       Running             0          1d
dotmesh-etcd-cluster-p8bj9lt82q                     1/1       Running             0          1d
dotmesh-etcd-cluster-tqcnzwq7jl                     1/1       Running             0          1d
dotmesh-operator-67d6bc7c66-plgcc                   1/1       Running             0          1d
etcd-operator-7b8496785f-7tz6l                      1/1       Running             2          1d
server-ip-172-18-0-163.us-west-2.compute.internal   0/1       ContainerCreating   0          2s
server-ip-172-18-0-173.us-west-2.compute.internal   0/1       Terminating         0          4s
Here is what I see in master's journalctl :
10623]: INFO: 2018/08/13 16:06:53.960804 added entry <IP_ADDRESS> to weave-Q}x3s}=G;0?;)ye4bcpR/n^39 of e3aa0424-9f12-11e8-9787-022c98761ed4
10623]: [158B blob data]
10623]: curl: (22) The requested URL returned error: 404 Not Found
10623]: ZFS is not installed on your docker host, and unable to find a kernel module for your kernel: 3.10.0-693.21.1.el7.x86_64
10623]: Please create a new GitHub issue, pasting this error message, and tell me which Linux distribution you are using, at:
10623]:
10623]:     https://github.com/dotmesh-io/dotmesh/issues
10623]:
10623]: Meanwhile, you should still be able to use dotmesh if you install ZFS manually on your host system by following the instructions at http://zfso
10623]:
10623]: Alternatively, Ubuntu 16.04 and later comes with ZFS preinstalled, so using that should Just Work. Kernel modules for Docker for Mac and other
10623]: + echo 'ZFS is not installed on your docker host, and unable to find a kernel module for your kernel: 3.10.0-693.21.1.el7.x86_64'
10623]: + echo 'Please create a new GitHub issue, pasting this error message, and tell me which Linux distribution you are using, at:'
10623]: + echo
10623]: + echo '    https://github.com/dotmesh-io/dotmesh/issues'
10623]: + echo
10623]: + echo 'Meanwhile, you should still be able to use dotmesh if you install ZFS manually on your host system by following the instructions at htt
10623]: + echo
10623]: + echo 'Alternatively, Ubuntu 16.04 and later comes with ZFS preinstalled, so using that should Just Work. Kernel modules for Docker for Mac an
10623]: + exit 1
ped libcontainer container cfdd9485adad3270e44ee911414e1766abb16e382641c037ea9134d7470b7662.
[10018]: systemdhook : cfdd9485adad: Skipping as container command is /require_zfs.sh, not init or systemd
]: umounthook : cfdd9485adad: only runs in prestart stage, ignoring
10623]: time="2018-08-13T16:06:54.074017248Z" level=error msg="containerd: deleting container" error="exit status 1: "container cfdd9485adad3270e44ee9
10623]: time="2018-08-13T16:06:54.074900982Z" level=warning msg="cfdd9485adad3270e44ee911414e1766abb16e382641c037ea9134d7470b7662 cleanup: failed to un
10623]: DEBU: 2018/08/13 16:06:54.521433 EVENT UpdatePod {"metadata":{"creationTimestamp":"2018-08-13T16:06:51Z","labels":{"dotmesh.io/role":"dotmesh-s
10623]: INFO: 2018/08/13 16:06:54.521468 deleting entry <IP_ADDRESS> from weave-+FPdY83%9%Hgj1MVl}gh(=3fA of e3a87a92-9f12-11e8-9787-022c98761ed4
Please let me know how to proceed.


Baseline is failing for analytical platform data development account

Expected Behaviour
Baseline to run successfully
Actual Behaviour
Error: Error creating CloudTrail: InsufficientSnsTopicPolicyException: SNS Topic does not exist or the topic policy is incorrect!

  with module.baselines.module.cloudtrail.aws_cloudtrail.cloudtrail,
  on .terraform/modules/baselines/modules/cloudtrail/main.tf line 11, in resource "aws_cloudtrail" "cloudtrail":
  11: resource "aws_cloudtrail" "cloudtrail" {

Steps to Reproduce the Problem

Other information
https://github.com/ministryofjustice/modernisation-platform/runs/3423084163?check_suite_focus=true

This is a bug, it happened on the tariff environment as well


Docker Images not working ?

I am looking to setup a docker compose based set up of evidence
Not looking to have a dev environment, however could not find any non-development docker image, so using this development image.
Pls help to get this docker container working.
Steps To Reproduce
Using below simple docker-compose.yaml
version: "3.7"
services:
  evidence:
    image: evidencedev/devenv:latest
    container_name: evidence
    init: true
    volumes:
      - ${pwd}:/evidence-workspace
    ports:
      - 3000
    depends_on:
      - db
    links:
      - db

# ..... rest is ignored for brevity
# db works fine as I have been able to connect to it using other applications.

Environment

Node version (node -v): N/A
npm version (npm -v): N/A
OS: Ubuntu LTS
Browser: N/A
Database: Mysql

Expected Behavior
Docker image starts up without issues.
Actual Behaviour
Container terminates / exits with below in logs
15/09/2023 10:32:25 am
Starting Evidence.dev development environment mounted on /evidence-workspace in the container.
15/09/2023 10:32:25 am
Provided arguments =>
15/09/2023 10:32:25 am
Running command => npm install && npm run dev -- --host <IP_ADDRESS>
15/09/2023 10:32:26 am
15/09/2023 10:32:26 am
up to date, audited 1 package in 180ms
15/09/2023 10:32:26 am
15/09/2023 10:32:26 am
found 0 vulnerabilities
15/09/2023 10:32:27 am
npm ERR! Missing script: "dev"
15/09/2023 10:32:27 am
npm ERR!
15/09/2023 10:32:27 am
npm ERR! To see a list of scripts, run:
15/09/2023 10:32:27 am
npm ERR!   npm run
15/09/2023 10:32:27 am
15/09/2023 10:32:27 am
npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2023-09-14T21_32_27_003Z-debug-0.log
Container stopped

@accforgithubtest if you are not looking for a dev container, are you looking for a container to host the evidence site ? Or to build it ?

I am not looking for a development container, just looking to self host evidence. But looks like there is only dev containers available.
Is there any other container I should be using instead ? Apologies if I am missing something here.

same question here