KAFKA-8531: Change default replication factor config

Call for review @cadonna @ableegoldman

As test against older broker versions, I started https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4463/ -- maybe we need to update some system tests...

Was also wondering about a potential error message -- not sure atm what error message a user would get if they run against 2.3 brokers and if the error message would be clear. Should we do anything about it?

Was also wondering about a potential error message -- not sure atm what error message a user would get if they run against 2.3 brokers and if the error message would be clear. Should we do anything about it?

Not sure either -- maybe you can use the soak test to spin up brokers on 2.2 against this PR and check out the error message + stack trace? I definitely think we should try to catch the error and log a more helpful error message (eg in 3.0 we changed the default replication factor to -1 which requires brokers to be on 2.3+, please update your brokers or manually set the replication factor config)

The logs show the following error:
[2021-04-13 15:52:17,685] ERROR stream-thread [main] Unexpected error during topic creation for streams-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition.
Error message was: org.apache.kafka.common.errors.UnsupportedVersionException: Creating topics with default partitions/replication factor are only supported in CreateTopicRequest version 4+. The following topics need values for partitions and replicas: [streams-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition] (org.apache.kafka.streams.processor.internals.InternalTopicManager:446)
[2021-04-13 15:52:17,687] ERROR stream-client [streams-wordcount-4b9ea556-bc49-4671-83e8-4d2adaba3677] Encountered the following exception during processing and the registered exception handler opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.  (org.apache.kafka.streams.KafkaStreams:484)
org.apache.kafka.streams.errors.StreamsException: Could not create topic streams-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition.
...
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Creating topics with default partitions/replication factor are only supported in CreateTopicRequest version 4+. The following topics need values for partitions and replicas: [streams-wordcount-KSTREAM-AGGREGATE-STATE-STORE-0000000003-repartition]

Do we think this would be sufficient? Or should we try to handle this specific UnsupportedVersionException explicitly?

Yikes -- no, I think we definitely need to handle that ourselves. I don't think users will have any idea what that means -- for one thing they'll wonder why they're suddenly seeing it now, and probably report it as a bug if we don't explicitly tell them that we changed the default replication factor. For another, how are they supposed to know how to interpret Creating topics with default partitions/replication factor are only supported in CreateTopicRequest version 4+ -- it doesn't even clearly state that the problem is older brokers, much less what version specifically the brokers need to be

Updated the PR.

I started a quick KIP for this change -- will wait until the KIP is accepted before merging.


Version 0.14.x removed -latency-ms

when starting server it gives an error
unrecognised option --latency-ms

Fixed in    https://github.com/MajorMJR/factorio-server-manager/commit/f9694bb6bcfe15095104390f41e0c3dfb66b619d
I've made the latency option optional, if the user selects a different value in the UI then the server will be started with the --latency-ms option.
I tested this with the latest 0.14.4 server version.  From the changelog they removed this option since 0.14, multiplayer performance seems to be improving nicely.

Ok that is probably a better solution than I had hacked together.  However the --latency-ms option appears to be removed from the arguments altogether.
`General options:
-h [ --help ]                 display help
--version                     show version information
-v [ --verbose ]              enable verbose logging
-c [ --config ] PATH          config file to use
--no-log-rotation             don't rotate log file
--mod-directory PATH          Mod directory to use
Running options:
-s [ --map2scenario ] arg     map to scenario conversion
-m [ --scenario2map ] arg     scenario to map conversion
--apply-update arg            immediately apply update package
--create FILE                 create a new map
--map-gen-settings FILE       Map generation settings for use with --create. See data/map-gen-settings.example.json
--start-server FILE           start a multiplayer server
--start-server-load-latest    start a multiplayer server and load the latest available save
--until-tick TICK             run a save until given map tick
--mp-connect ADDRESS          start factorio and connect to address
--load-game FILE              start Factorio and load a game in singleplayer
--benchmark FILE              load save and run benchmark
--benchmark-ticks N (=1,000)  number of ticks for benchmarking. Default is 1000
--force-opengl                use OpenGL for rendering
--force-d3d                   use Direct3D for rendering
Server options:
--autosave-interval N (=2)    server autosave interval, in minutes
--autosave-slots N (=3)       server autosave slots
--allow-commands arg (=true)  allow (true), disallow (false), or restrict (admins-only) use of the command console
--no-auto-pause               don't automatically pause when no players are
connected
--port N                      network port to use
--rcon-port N                 Port to use for RCON
--rcon-password PASSWORD      Password for RCON
--server-settings FILE        Path to file with server settings. See data/server-settings.example.json`
On an unrelated note, is it possible to add the server default port, autosave interval, and autosave number to the config file rather than hardcoded into ServerCtl.jsx?

I'm not sure if i'm doing soemthing wrong... but I get this error while trying to execute the binary
`
pickle@s:/factorioServers/factorio-server-manager$ ./factorio-server-manager --dir /factorioServers/factorio --port 8888
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x5fe43e]
goroutine 1 [running]:
panic(0x938080, 0xc8200100c0)
/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
github.com/syndtr/goleveldb/leveldb.(*DB).setClosed(0x0, 0xc91d00)
/home/chicks/go/src/github.com/syndtr/goleveldb/leveldb/db_state.go:220 +0x4e
github.com/syndtr/goleveldb/leveldb.(*DB).Close(0x0, 0x0, 0x0)
/home/chicks/go/src/github.com/syndtr/goleveldb/leveldb/db.go:1027 +0x58
github.com/apexskier/httpauth.NewLeveldbAuthBackend(0xc820011a60, 0xc, 0x0, 0x0, 0x0, 0x7f8405e75028, 0xc820011ac0)
/home/chicks/go/src/github.com/apexskier/httpauth/leveldbAuthBackend.go:36 +0x28d
main.(*AuthHTTP).createAuthDb(0xc82000abe0, 0xc820011a60, 0xc, 0x0, 0x0)
/home/chicks/factorio-server-manager/src/auth.go:30 +0x81
main.main()
/home/chicks/factorio-server-manager/src/main.go:85 +0x147
`
Perhaps I am just compiling this wrong?

Did you build the binary or is it the compiled one in the release?  It looks like there is some panic getting generated when it is trying to create the leveldb files.  Maybe check if there are write permissions in the directory you are starting the server manager from.

I compiled it from the source as there isn't a compiled version for factorio 0.14.
I chmod everything with 777 perms to make sure it wasn't a permissions issues.
Though, I did compiled this on a different machine. and the error is refering to "/home/chicks" which doesn't exist on my server i am trying to execute, but that shouldn't be a problem?

Check your conf.json file, maybe there is path defined there for database_file?  Just guessing since the server is looking in the /home/chicks directory on startup.

{
    "username": "admin",
    "password": "factorio",
    "database_file": "auth.leveldb"
}

Looks like the default config.json
Oh.. .umm just took a wild guess, and deleted the auth.leveldb directory, and it works fine now...
strange. Well, thanks anyway.

Awesome I guess that's a workaround unless we can find what caused database files to crash the program.

Yeah, i didn't think to copy the files before deleting them...

any chance to upload the latest binary for 0.14.
i tried to build it for 2 days in my ubuntu, it is always missing files from github.

@NQRott Yeah, i had the same problem. I getting those.
Use go get "url" replace url with the github url it says is missing.

Yeah you will have to 'go get' all the third party libraries, check the .travis.yml file for a list of the libraries URL's.
I'll build a release tonight when I get home this evening after work.

thx now i can start the server. it was compiled with the wrong name. i just replaced it with the old binary.

I've uploaded the latest release with the bug fixes for 0.14.
https://github.com/MajorMJR/factorio-server-manager/releases/latest


Redefining Setpoint Classifications: Target and Threshold

How do we define setpoint? I personally like Wikipedia's definition:

The desired or target value for an essential variable, or process value (PV) of a control system which may differ from the actual measured value of the variable.

Let's call this a "Target Based Setpoint"
Target-Based
These are values that is the desired target for a process variable (PV). Examples include:

Temperature Setpoint: A value specifying the desired temperature to be maintained.
Humidity Setpoint: A value specifying the desired humidity to be maintained.
Differential Setpoint: A value specifying the desired difference between two points in a system.

However, there are many terms used to describe different types of setpoints, such as:

Cutoff Setpoint
Reset Setpoint
Enable Setpoint

These setpoints are an attempt to describe something else.. and it doesn't fit into the definition mentioned above. So how do we distinguish these types of setpoints? I would like to propose a term, "Threshold Setpoint".
Threshold-Based
These values initiates a specific action when a process variable crosses a certain threshold. They can be further categorized into upper and lower thresholds:
Upper Thresholds
Initiates a specific action when the process variable exceeds specified value.

Cooling Temperature Setpoint: Activates cooling when temperature exceeds this value.
Dehumidification Setpoint: Activates dehumidification when relative humidity exceeds this value.
CO2 setpoint: Activates ventilation or other CO2 mitigation actions when CO2 concentration exceeds this value.

Lower Thresholds
Initiates a specific action when the process variable falls below specified value.

Heating Temperature Setpoint: Activates heating when temperature falls below this value.
Humidification Setpoint: Activates humidification when relative humidity falls below this value.

Example

Let's compare "Effective Cooling/Heating Setpoint" to "Effective Target Setpoint" as an example:

Effective Cooling Setpoint: Serves as an "Upper Threshold", which initiates the cooling process
Effective Heating Setpoint: Serves as a "Lower Threshold", which initiates the heating control process
Effective Target Setpoint: Serves as a Target Setpoint, serving as a desired temperature to be maintained.

Key Takeaway
Clearer setpoint classification is essential for understanding the nature of a setpoint. The distinction between target-based and threshold-based setpoints enhances our ability to represent all types of setpoints observed in building systems.
Action items
We could introduce some variation of the following:
brick:Target_Setpoint rdfs:subClassOf brick:Setpoint.
brick:Threshold_Setpoint rdfs:subClassOf brick:Setpoint.
brick:Upper_Threshold_Setpoint rdfs:subClassOf brick:Threshold_Setpoint.
brick:Lower_Threshold_Setpoint rdfs:subClassOf brick:Threshold_Setpoint.

Next, we can categorize all existing setpoint classes under the following superclasses.

Target Setpoint
Upper Threshold Setpoint
Lower Threshold Setpoint

Note: I don't have a strong opinion on "Threshold Setpoint" and I am open to other suggestions.

Discussion

Deadband Setpoints are neither target-based or threshold-based, rather they determine the distance between two thresholds. Is this a setpoint, or should it be moved to the parameter class? (I suggest it is moved to parameter class)
Time Setpoint is also not target based or threshold based. Is this a setpoint, parameter, or something else?
This is a controversial topic, but it could also be argued that the concept of "Threshold Setpoint" isn't actually a setpoint.

I like the additional refinement, even though I prefer a shallow hierarchy rather than a deeper one, and the new names are better.
I think of deadband as an additional parameter to a threshold calculation, so in the case where a deadband is manipulated to allow for "tight" or "loose" control there might be some additional range that the deadband is expected to vary between, but is there going to be an alarm when the deadband exceeds some value?
I thought Time Setpoints where things like minimum/maximum on/off time, but looking at the version 1.3 hierarchy they are something different.

Hi @gtfierro,
I'd like to move forward with this PR, but want to get your feedback first. Here are a few options we could consider:
1. Add additional parents to each Setpoint
In this approach, we would avoid restructuring existing class hierarchies and add one of the following classes as a parent:

Target Setpoint
Upper Threshold Setpoint
Lower Threshold Setpoint

setpoints = {
    "Setpoint": {
        "subclasses": {
            "Target_Setpoint": {},
            "Threshold_Setpoint": {
                "subclasses": {
                    "Upper_Threshold_Setpoint": {},
                    "Lower_Threshold_Setpoint": {},
                }
            },
            "Pressure_Setpoint": {
                BRICK.hasQuantity: BRICK.Pressure,
                "parents": [
                    BRICK.Target_Setpoint,
                ],
                "subclasses": {}
            },
            "Cooling_Temperature_Setpoint": {
                BRICK.hasQuantity: BRICK.Temperature,
                "tags": [TAG.Point, TAG.Temperature, TAG.Setpoint, TAG.Cool],
                "parents": [
                    BRICK.Upper_Threshold_Setpoint,
                ],
                "subclasses": {}
            },
            "Heating_Temperature_Setpoint": {
                BRICK.hasQuantity: BRICK.Temperature,
                "tags": [TAG.Point, TAG.Temperature, TAG.Setpoint, TAG.Heat],
                "parents": [
                    BRICK.Lower_Threshold_Setpoint,
                ],
                "subclasses": {},
            },
            # etc.
        }
    }
}

2. Add a new tag to each Setpoint
setpoints = {
        "subclasses": {
            "Pressure_Setpoint": {
                BRICK.hasQuantity: BRICK.Pressure,
                 "tags": [TAG.Target, TAG.Pressure, TAG.Setpoint],
                "subclasses": {}
            },
            "Cooling_Temperature_Setpoint": {
                BRICK.hasQuantity: BRICK.Temperature,
                "tags": [TAG.Upper_Threshold, TAG.Point, TAG.Temperature, TAG.Setpoint, TAG.Cool],
                "subclasses": {}
            },
            "Heating_Temperature_Setpoint": {
                BRICK.hasQuantity: BRICK.Temperature,
                "tags": [TAG.Lower_Threshold, TAG.Point, TAG.Temperature, TAG.Setpoint, TAG.Heat],
                "subclasses": {},
            },
            # etc.
        }
    }
}

3. Restructure class hierarchy (Preferred)
Existing class hierarchies are altered, but nearly all class names remain. (See notes below)
Though this approach requires the most changes, it addresses the issue of incorrect semantics by clearly defining each class within its respective Setpoint type. This eliminates ambiguity and ensures that each class accurately represents its intended function.
.
├── target_setpoint
│   ├── pressure_setpoint
│   ├── humidity_setpoint
│   ├── temperature_setpoint
│   └── ...
└── threshold_setpoint
    ├── lower_threshold_setpoint
    │   ├── heating_setpoint
    │   ├── humidification_setpoint
    │   └── ...
    └── upper_threshold_setpoint
        ├── co2_setpoint
        ├── cooling_setpoint
        ├── dehumidification_setpoint
        └── ...

Some superclasses such as Temperature_Setpoint currently contain both Target and Threshold Setpoints. I propose repurposing Temperature Setpoint to be exclusively for "(Target) Temperature Setpoints". Alternatively, we could deprecate Temperature_Setpoint and replace it with Target_Temperature_Setpoint.

cc @jbkoh

Options 1 and 3 feel like duals. They both organize the setpoints along 2 dimensions: by quantity (e.g., Temperature) and by the setpoint type (e.g., Threshold). The difference between them is whether we organize the Python files by quantity or by setpoint type. Do I understand this right?


RISC-V: added rsync to image

Updated image for nightly testing for opencv/ci-gha-workflow#165

Updated image has been pushed to the registry, rsync is available now.


Issue 131

added flag or optional environment variable for the environment name.
This should be used in the protocol names to segment the network appropriately between dev & test.

I also incremented the version number to v0.0.5-alpha


update dockerfile according to the new INSTALL.md

From https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/INSTALL.md:

additionally install tqdm, cudatoolkit, apex
add FORCE_CUDA argument

Note that the argument CUDA is re-declared, this is done so that its value can be reused, following the instruction here.

cc @miguelvr

LGTM :+1:


Organize wiki

It's kinda messy right now...

+1
@lhorie : "yeah I was thinking we could have three pages: one for projects consuming mithril, one for projects that complement mithril, and one for links to snippets and patterns" April 27 2015 10:55 AM
Several links on the Community page would need to be transferred to one of the above pages and the links from the spreadsheet distributed accordingly.
Tools that complement Mithril but are not part of userland source code:

Sublime/Vim/Textmate/Atom snippets
Sweet.js macros
pre-processors (MSX, jss/mjss)

Projects consuming mithril (with accessible source code) :

todomvc
lichess (UI)
the various starter kits

Snippets and patterns would preferably link to working examples on jsfiddle/jsbin/plunker etc.

Recommended tutorials should be collected on the Mithril home page, but there might also be a page for "secondary" tutorials, boilerplates, starter kits (Further Learning ?). The good ones might graduate to the home page when mature.

@lhorie The wiki home page needs an intro that you should write. Something about community contributions and "what you'll find here" ?

I'd be happy to contribute to this effort. With the merging of the components branch and the crazy interest coming from Japanese Twitter, I think organizing learning materials should be a high priority.

Recommended tutorials should be collected on the Mithril home page, but there might also be a page for "secondary" tutorials, boilerplates, starter kits (Further Learning ?). The good ones might graduate to the Mithril home page when mature.

I really like this idea

:+1:

FYI, I started to organize it. Anyone wants to start putting the stuff in the uncategorized section (especially the items in the google spreadsheet) into their appropriate sections?

Gotta do some paid work. I'll work on it later if no one beats me to it.

Spreadsheet links copied to wiki page.
Done.
@lhorie Could you look over the uncategorized section and find homes for those things. The spreadsheet link can be removed. I didn't want to be so bold ;)

@lhorie I was thinking, now that events (meet-ups, periscope streams, talks) are happening, You might want to reserve a place for them on the Mithril home page.
Same goes for Mithril in-the-news. You commented that you missed a long Reddit post recently. The Twitter links at the bottom of the home page are very stale.
I think the footer could use a makeover.


Exception from ElasticSearch Write Module in DataFlow

I am seeing the following exception being thrown from the ElasticSearch Write module when running my streaming pipeline from Google DataFlow. I am using Apache Beam 2.39.0. I hope someone will be able to help me or tell me if this is a bug. Thanks a lot.

2022-06-02T02:21:07.023945713ZError message from worker: java.lang.ClassCastException: class org.apache.beam.sdk.transforms.windowing.GlobalWindow
cannot be cast to class org.apache.beam.sdk.transforms.windowing.IntervalWindow (org.apache.beam.sdk.transforms.windowing.GlobalWindow
and org.apache.beam.sdk.transforms.windowing.IntervalWindow are in unnamed module of loader 'app') org.apache.beam.sdk.transforms.windowing.IntervalWindow$IntervalWindowCoder.registerByteSizeObserver(IntervalWindow.java:142)
org.apache.beam.sdk.coders.IterableLikeCoder.registerByteSizeObserver(IterableLikeCoder.java:209) org.apache.beam.sdk.coders.IterableLikeCoder.registerByteSizeObserver(IterableLikeCoder.java:59)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:640)
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.registerByteSizeObserver(WindowedValue.java:558)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$ElementByteSizeObservableCoder.registerByteSizeObserver(IntrinsicMapTaskExecutorFactory.java:403)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputObjectAndByteCounter.update(OutputObjectAndByteCounter.java:128)
org.apache.beam.runners.dataflow.worker.DataflowOutputCounter.update(DataflowOutputCounter.java:67)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:43)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:285) org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:251)
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:85)
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnFinishBundleArgumentProvider$Context.output(SimpleDoFnRunner.java:309)
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner$DoFnFinishBundleArgumentProvider$Context.output(SimpleDoFnRunner.java:304)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$BulkIO$BulkIOBaseFn$FinishBundleContextAdapter.output(ElasticsearchIO.java:2415)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$BulkIO$BulkIOBaseFn.flushAndOutputResults(ElasticsearchIO.java:2422)
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$BulkIO$BulkIOBaseFn.finishBundle(ElasticsearchIO.java:2382)

Imported from Jira BEAM-14551. Original Jira may contain additional context.
Reported by: Declan.

I was able to repro on Dataflow today.  I would guess this issue is widespread for any pipeline with windowing and Sinking data to ES.  This is a regression introduced by https://github.com/apache/beam/pull/17112

Does #17112 completely solve it? Can it be closed?

Could we move this to 2.41.0? Does this need to be a blocker for 2.40.0 ?

The ElasticsearchIO connector is completely broken in 2.39.0 as a result of this issue. I would say it should be a blocker to either fix this or revert  https://github.com/apache/beam/pull/17112 and try again to fix the issue that it was meant to patch.

+1 to reverting and not blocking the release.

is the issue that expand() uses setWindowingStrategyInternal ?  This doesn't create a new PCollection so in theory "propegates back" to the producer (your batching DoFn) and ends up with a mismatched coder. (I think at least?)

Thanks @steveniemitz , that very well could be the issue!  I definitely need some guidance.  I had seen in other places in the codebase that Flatten was used to explicitly force the creation of a new PCollection.  My original idea was to write a test so that I could play around with setWindowingStrategyInternal Vs. Window.into Vs. Flatten/PCollection.createPrimitiveOutputInternal but I wasn't able to reproduce this issue in testing.
Any thoughts on alternative methods of reapplying the same windowing without using setWindowingStrategyInternal?

That’s a good point. I’ve honestly debated whether or not an IO ought to preserve windows at all given that responses from a Sink could be viewed as effectively a new Source. But I suppose preserving windows could be useful for joining data sent to a sink and data returned from that sink via CoGroupByKey.
Back to your original idea of just using the windowing function, I’ll give that a try. Thanks!

Just confirmed that @steveniemitz suggestion of using the WindowFn + Window.into rather than setWindowingStrategyInternal corrects the error described here.  I must admit I don't fully grasp why yet (following up on @steveniemitz's comment RE setWindowingStrategyInternal not creating a new PCollection for my own understanding).
Opened a PR at https://github.com/apache/beam/pull/21895 as an alternative to reverting.

I think I will proceed with Revert if I can't get a confirmation that the fix will be the preferred route : /

I don’t know what all the decision criteria are for which is preferred. The proposed fix definitely corrects the error observed in Dataflow. I believe the fix is merely correcting a misunderstanding on my part that should have been included originally with #17112.
Reverting will reintroduce a potential data correctness bug. But I fully concede if the fix is deemed to simply not have had enough time for review and further testing.

ok I'll cut the branch without either and we can resolve in the coming days. Thanks @egalpin

Thanks @pabloem , apologies for the extra work that entails as release manager 🙏

aw no worries! I appreciate your help with this fix.

Ok so I did a little more digging for the sake of my own understanding and I think I have a much better grasp of what setWindowingStrategyInternal is used for.  It seems to be intended simply for cases where the programmer knows that the data contained in a new PCollection has a specific the same WindowingStrategy as a prior PCollection i.e. the data type and window type are known to be the same in a new PCollection as they were in a previously known PCollection.  Ex in Window.into[1].
setWindowingStrategyInternal is not meant to be used to actively restore i.e. re-apply a WindowFn, rather it's used to indicate that a previously applied WindowFn is already suitable.
I feel I have better clarity on why re-applying (i.e. Window.into(<priorWindowFn>)) resolves this issue.  I would like to recommend that the fix in https://github.com/apache/beam/pull/21895 be the preferred path forward.
[1] https://github.com/apache/beam/blob/29f30b3605824a47cbc68229f63af44a34cf0885/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java#L404-L408


feat: crda cli: Analyse command - npm

CRDA CLI: Analyse Command - NPM
Build: go build -o crda
Execute:
./crda analyse --file /absolute/path/to/package.json

Development Run
go run main.go analyse --file /absolute/path/to/package.json

Codecov Report

Merging #25 (8aa5ae2) into main (71f62bd) will increase coverage by 1.60%.
The diff coverage is 70.83%.

@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
+ Coverage   50.00%   51.60%   +1.60%
==========================================
  Files          10       11       +1
  Lines         288      312      +24
==========================================
+ Hits          144      161      +17
- Misses        128      132       +4
- Partials       16       19       +3

Impacted Files
Coverage Δ

analyses/npm/matcher.go
70.83% <70.83%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 71f62bd...8aa5ae2. Read the comment docs.


How do i give you money

This project is great and I would like to fund some beer for you :D

Haha that's very kind of you, let's say I'll let you invite me for one if one shiny day we coincide in SF, Budapest, or anywhere else in the world. 🙂

Haha sounds delightful. Budapest is beautiful; I've visited a couple of times.
But seriously if you ever put a donate button here ping me.


Solve all the eslint-disable issues

Description
Solve all the different types of issues hinted by the linter and remove comments.
Context
ESLint is meant to point out possible issues and anti-pattern. This is often done due lack of further implementation, wrong implementation and temporary "fixes".
As we have at the time 213 occurrences, with this issue we want to tackle all down and define in separated cases or rather solve them all at once when possible.
Concisely most common, may not be complete, list of cases:

Unhandled logs: Can be replaced with helper
Typescript misconfiguration: Can be updated
Case issues: can be defined in the linter
Actual bad practices leading to bugs: Use of let over cons, etc

Linking #3563


Run prelude.rs after init.evcxr if one exists. Fixes #79

Tested manually (both with and without the prelude), since there aren't automated tests for this I could find (and it seems annoying to test without something like #78).

Ah, I missed that you said you'd accept #78, one minute and I'll add a commit for that too (since otherwise it would conflict).

Note: in case it matters, I tested with evcxr_repl, I'm not sure how to set up the jupyter stuff to test evcxr_jupyter, but I have no reason to think it wouldn't work (but I am willing to try if you think it's needed).

Thanks!


Task: rename this repo to kebab-case

I named this repo wrong, all the rest use - not _.
This will also involve updating S3 push role here:
https://github.com/WFP-VAM/seasmon-aws/blob/1c4323a01baefc230363dcc5ff7e5433d70bc862/cf-templates/svc_iam.yaml#L45
Unless we want a completely different name that is.

we will need a different name, as we do for seasmon-aws

Have we settled on hdc-algo for this repo?

closed with #36 🎉


Assignment 3 - mini project

Screenshots and DDL  statements:
https://www.notion.so/Mini-Project-b8585f88a5594b188f2acbda5d3fda3f
Video link:
https://drive.google.com/file/d/1JK7hZ5NuL_QAeALB5FmcDLpX-20hJhBb/view?usp=sharing

Metric
Value

alert_status
:white_check_mark: OK

bugs
0

code_smells
88

reliability_rating
A (0 Bugs)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0

Metric
Value

alert_status
:x: ERROR

bugs
0

code_smells
54

reliability_rating
A (0 Bugs)Note: A being best, E being worst

security_rating
A (0 Vulnerabilities)Note: A being best, E being worst

vulnerabilities
0

please note that the remaining code sniffs are consequences of the test cases and expected naming conventions


Older Rubies (2.1 to 2.3 at least) don't install on default macOS image (10.13.6)

I've described this issue here but will post it here as well since I believe it belongs here as well. Feel free to close this issue if this is not the place. I'm not sure if it's a Travis CI issue or if I should open an issue in the rvm repo.
It seems a new OpenSSL version was released on Homebrew (1.1, from 1.0 (this also contains a successful and a failed test run)) and also the macOS build image on Travis changed since the last test run of jaro_winkler (check also the available Rubies etc).
That seems to be the cause for the failures.
Somebody posted a possible workaround to get an older OpenSSL version installed via homebrew.
Further reading:
https://travis-ci.community/t/rvm-install-fails-on-osx-due-to-missing-libraries/5596
https://github.com/rvm/rvm/issues/4819
https://github.com/rvm/rvm/issues/4781
https://github.com/rvm/rvm/issues/4463
https://github.com/Homebrew/homebrew-core/issues/46454

I am not sure if we can reliably support Rubies <= 2.3.x on macOS at this point.
The problem is quite complicated:

Rubies before 2.4.0 do not compile with OpenSSL 1.1, thus we need OpenSSL 1.0.x, which is no longer supported.
Homebrew, our package manager on macOS, has dropped support for OpenSSL 1.0.x, justifiably.
This means that we have to rely on a third-party tool (or create our own) to provide OpenSSL 1.0.x. It is possible to compile OpenSSL 1.0.x on the fly, but it will take quite some time to do so. I have not found any binary package to sidestep this issue (let alone a trustworthy one that we can use long term).
RVM currently is not aware of this issue, so it appears to prefer the latest OpenSSL version when we ask for Ruby < 2.4.0.
The problem would be mitigated if we could compile a static macOS binary that can be ported across machines, but as far as I know, currently there is no way to do so.

It is conceivably possible to:

Compile OpenSSL 1.0 (this is not difficult)
Coerce RVM to compile Ruby with this OpenSSL instead of the latest version
Package all the binary files into the archive, so that users don't have to compile things at build time.
???
$$ Profit $$

As I mentioned in my mail to support, there are at least two ways to do so:
# using rvm pkg install (deprecated)
rvm pkg install openssl
rvm install 2.3.0 --with-openssl-dir=$HOME/.rvm/usr

or
# alternative using rbenv recipe
brew install<EMAIL_ADDRESS>rvm install 2.3.0 --with-openssl-dir=$(brew --prefix<EMAIL_ADDRESS>

@hurricup Do you have a URL to the working build?

No. This is how I set up locally. But travis for some reason can't do the same. And this is really strange.

With
language: generic
os: osx
osx_image: xcode10.3
before_install:
  - env HOMEBREW_NO_AUTO_UPDATE=1 brew install<EMAIL_ADDRESS>  - rvm reinstall 2.3.0 --with-openssl-dir=$(brew --prefix<EMAIL_ADDRESS>install:
  - true
script:
  - "ruby -v -ropenssl -e \"puts OpenSSL::VERSION\""

You can compile OpenSSL 1.0 and Ruby 2.3.0
This takes about 30 minutes.

I presume travis could pre-build them and use like binary distribution. I know it's kinda tricky on macos, but still possible.

In theory, yes, but I have not found a way to play nicely with Homebrew.

As in, bundling up what Homebrew writes on the build machine in such a way that can be reused elsewhere. Some of it might be simulated by caching, but it seems to fail in a very hard-to-understand ways.
Ruby 2.4 support is already dropped. I question the benefit of continuing to support 2.3, to be honest.

The question is not about lifetime from devs, but usage statistics. Like https://stats.rubygems.org/

As in, bundling up what Homebrew writes on the build machine in such a way that can be reused elsewhere.

Since this is RVM's problem, you either supplement its logic in travis-build (brew install before rvm install), tap into its logic (I don't know how you tap into it to use https://rubies.travis-ci.org/ since the relevant logic is not on Github so can't say) -- maybe something similar, or make a PR against it to fix the problem for everyone.
Alternatively, you can use rbenv for affected versions who have already fixed that problem for themselves (I don't know how compatible their Ruby installation is though).


no vertical cursor progression on GLFs (sixel)

Contour Terminal version
<IP_ADDRESS>
Installer source
GitHub: release page
Operating System
Ubuntu 18 LTS
Architecture
x86-64
Other Software
No response
Steps to reproduce
Plz see https://github.com/jerch/xterm-addon-image/issues/37 for details on the issue. There is also this testscript done by @hackerb9 to check TE behavior.
Expected Behavior
GLFs from sixel should advance the text cursor to the bottom, if sixel scrolling is on.
Actual Behavior
Currently there is no text cursor progression from multiple GLFs.
Additional notes
(@Utkarsh-khambra  This is the follow-up from #820.)

Looks like you want the ANSI text cursor to be placed on the next line at the beginning of the image's column. I'm sure I was NOT reading that in the spec, but I'm kinda blind... so I might have misread or skipped that by accident, I guess.

I tried to document the correct cursor positioning in the test case here:
https://github.com/hackerb9/vt340test/blob/main/j4james/cursor_position.sh#L3-L8

On a genuine VT340, after an image is output, the text cursor position is set to the same row as the top of the final sixel line (regardless of whether anything is output on that line). The column position remains unchanged, i.e. it should align with the left of the image.

Note that the "top" part of that statement is important when it comes to supporting aspect ratios other than 1:1, because a sixel line can end up spanning multiple text lines.

Looks like you want the ANSI text cursor to be placed on the next line at the beginning of the image's column. I'm sure I was NOT reading that in the spec

Well it is phrased somewhat fuzzy in DEC 070, but here is the relevant paragraph (copied from pdf, so it may contain weird symbols or broken/scrambled words):

11.3 Exiting SIXEL MODE
The Horizontal Active position of the ANSI text is not affected by Sixelel mode and set
exactly to what is was when Sixel mode was entered.

NB: Thats the col direction indentation - col stays where sixel started, no matter what.

The vertical active position of the ANSI text is affected by Sixel mode. The ANSI text
vertical position is updated to reflect the vertical motion commands executed in the Sixel
printing. Note that on line oriented devices, this may leave the ANSI text active position
with a non integer line value, only getting back on line boundaries after an absolute
vertical position command. Also, the vertical positioning commands of Sixel mode use the
ACTUAL grid size, which may be different than the specified grid size. Therefore, software
doing pagination should issue an absolute position command before placing characters to
insure placement as expected.
Some devices may require re-alignment to a grid established for ANSI text after leaving
Sixel mode. In this case, an adjustment less than one character height may occur in a
downward direction. Conforming software will not assume any exact alignment of the
ANSI text vertical position following a Sixel image.

Thats the fuzzy part for vertical progression (tried to highlight the imho important parts). Note that the test of proper GLFs alignment as on a vt340 is not strictly enforced by DEC 070 (last sentence). The matter with DEC 070 is furthermore complicated by vt240 with its "I dont care for text progression at all", which also is not directly reflected by 070 rules.
Also the "top part" behavior discovered on the vt340 cannot really be read from these lines, but it somewhat makes sense, if one sees the sixel cursor to be placed at the first pixel row and "clipping" to text position wherever that first pixel row of the last sixel band ended up (just my interpretation, might be wrong).

Note that the test of proper GLFs alignment as on a vt340 is not strictly enforced by DEC 070 (last sentence).

Yeah, but I don't think the STD-070 documentation is a great reference for sixel, since it seems to be mostly targetting their printers rather than their terminals. There is no mention of the VT340, and some things described there simply weren't applicable to the terminals (e.g. grid sizes), or behaved quite differently (e.g. aspect ratios).
So if you're just going with what is documented in STD-070, you can basically do whatever you want. But you're not going to get much interoperability that way. Back when I was testing the various sixel implementations I think found eight different ways in which modern TEs were interpreting the cursor position! And that's not even counting the different modes that some people invented.

The matter with DEC 070 is furthermore complicated by vt240 with its "I dont care for text progression at all", which also is not directly reflected by 070 rules.

That is actually described in the "deviations" section at the end of the chapter.
I should also add that the final text cursor position is only one aspect of this issue. You also need to consider scrolling, i.e. when and how a sixel image will cause the viewport to scroll. The algorithm for that is quite different from the text cursor positioning, because the image can overflow the viewport before the text cursor would.
There is a separate test case that demonstrates that behavior here:
https://github.com/hackerb9/vt340test/blob/main/j4james/scrolling.sh
And for more info on how exactly it's supposed to work, it's probably best to look at the issue where we figured out all the details.
https://github.com/hackerb9/vt340test/issues/11


Move TeraEd to new facade project

Also replaces the @EditorRange annotation with the more general @Range annotation. Maybe it could be moved outside the NUI package at some point of time.

Refer to this link for build results (access rights to CI server needed):
http://jenkins.terasology.org/job/TerasologyPRs/305/
Hooray Jenkins reported success with all tests good!

Works! Kinda :-)
I was able to tweak at the sky shader dynamically while in-game, making this dark and dreary, yet also sort of sunny scene:

Ping @emanuele3d - new toy! If you haven't had a chance to play with it much before. I made a run config for it in IntelliJ so it is easy to start. Just pick one of the shaders from the window menu to expose properties to play with.
I am not sure they all work, and the window doesn't dispose when you try to exit. Nor does it seem to respect -homedir, it used my local OS save dir instead. But it has been a long time since it was looked at so I'm sure it needs some more attention to behave again.
But this is a good first step! :-)

Ping acknowledged!


import kotlin.String necessary?

Since 4.0.2 an import to kotlin.String is created in the Versions.kt.
I am not sure but suspect, that this import is redundant. So maybe skip it?

The KotlinPoet has unit tests check that the lines import java.lang.Object or import kotlin.Boolean are present.
https://github.com/square/kotlinpoet/blob/485a9b5cc3fe09b9e422b97926fa120ad9031450/kotlinpoet/src/test/java/com/squareup/kotlinpoet/AnnotationSpecTest.kt#L323-L324
I don't really know why they decided to add those redundant imports and am too lazy to check that with them.

Reason stated by KotlinPoet in the issue opened by LouisCAD:

Unfortunately they're necessary because if you define a class with a conflicting name in your own package it will take precedence.

So those imports are here to stay


Add options in GUI/Menubar for what to show in the titlebar

Some users may not care about what index the image has in the folder or what resolution it is.

fixed with https://github.com/lambdan/imageviewer5/commit/b719110ac7487a1daf360e99f2dcb5c27010b76e


Codechef Chess Match in C++

Problem link : https://www.codechef.com/START5C/problems/BLITZ3_2
Programming language

[ ] C
[x] C++
[ ] Java
[ ] Python

Additional context: LGM-SoC'21 participant

Please assign this to me @ravikr126 @Saurabh4626

@tanvi355 Can I work on it?

This issue is already solved


docs: adjust difficulties and sort exercise order

The diff might be confusing.
Votes have been converted to bins of easy, middle, hard.
I sorted the exercises by difficulty first and then by name, as Jeremy said it is the intention for V3.

Should we keep a copy of the code for sorting the exercises in the repo? Then we can rerun it as needed.

Should we keep a copy of the code for sorting the exercises in the repo? Then we can rerun it as needed.

I did it manually, but I'm pretty sure there is a smart jq one liner I could have used.


Black screen

Some time after killing pngview have an empty black screen on raspbian stretch lite 4.14
run XServer with nodm and Electron (like kiosk).
script to reproduce is like:
#!/bin/bash
while :
do
        echo "Run"
        /usr/local/bin/pngview -b 0 -x 376 -y 730 -n /home/pi/reacthome/assets/hangup.png &
        PID=$!
        sleep .03
        kill $PID
done

Just have try
sudo xinit xterm

and script above. Result is same.

Solved! Problem was in hardware video core


How to integrate with react-intl?

Razzle seems pretty nice for dev.  I'm pretty impressed with it.  But, how would you write such an app with internationalization and translation support?  This is something that has to interact pretty intimately with the server and client side rendering system, since you want to be able to produce server-side rendered content in any language but you only want to ship the messages for a particular language on the client side to keep things light.  I'd appreciate it if you had a guide on how to integrate this.

You should probably use a custom renderer to include your IntlProvider with the correct messages in the server.
In the client I would recommend using dynamic imports to be able to load the bundle. If you want to include the current locale messages directly you can include it as a global variable or with data.

Anyone else run into this? Code samples would be great


Adding CORS headers for GET API requests.

Coverage increased (+0.002%) to 92.169% when pulling f7e0ab4b9d920f714bbd468df45397db256220d7 on cors into 67c18a23c748411d0d86a4e99205bb1e24fe519c on master.


[Form Recognizer] Specific test failing in min-max only

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1353806&view=logs&j=1c9a204a-48a9-5154-5204-230e40dbc42f&t=80440deb-e0c4-5351-a2fc-e671cb2babbb
Looks like, started happening after the tests are migrated to the new recorder

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Azure.Core:0.30330625,Cognitive - Form Recognizer:0.27047768,Docs:0.055486377'

Closing since this is a flaky service issue
Proof with a clean green run on main - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1360230&view=results


Respond to empty fake responses

Earlier if the response text specified in the textarea was blank, such requests were responded with default text. This PR allows interceptor to mock empty fake responses

Closing this as this branch was not based on master.  Will open another PR


Create check-array-contains-another-array.md

Check if an array contains another array

Thank for the PR, @lupu60
Instead of creating new PR, can you add more method to this https://1loc.dev/#check-if-an-array-is-subset-of-other-array?
Here is the source.

I have opened another pr by mistake didn't read all, lol :)))  https://github.com/phuoc-ng/1loc/pull/342.
Should I close this one?


System.err.println in Bike2WeightEncoder

https://github.com/graphhopper/graphhopper/blob/master/core/src/main/java/com/graphhopper/routing/util/Bike2WeightFlagEncoder.java#L176
In my opinion this should be removed or changed to logger?

This is a could hint. The problem is: there could be several and this is somewhat unrelated to GraphHopper. See #435. Also we do writing to std err in other cases when this is not really a GraphHopper bug but should be fixed in OSM. Let us think about what to do best with these.

@karussell Ah thought this would be a typo. Well in this case, I like the idea of splitting the ways on import as described in #435.

Okay, if this is then fixed we should throw an error here or at least log as warn.
Still I'm unsure what to do if just OSM is erroneous and we cannot parse something e.g. in max speed. Or what if names are too long like described here and this logging is happening here

I guess fixing as much as possible on import is the best, report it as info or warn. Requiring to check if distance is infinite each time is not that nice in my opinion. What do you think?
BTW: It's crazy what problems arouse when working with OSM data. The russian names are crazy.

Requiring to check if distance is infinite each time is not that nice in my opinion

Yes, that is very ugly and we should definitely move the check to setDistance instead.


Error from pip install

Hi, so recently trying to install from pip install when I try running tensorboard from the command line I get:
RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa
Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/tensorboard/tensorboard.py", line 32, in <module>
    from tensorflow.python.summary import event_file_inspector as efi
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/python/summary/event_file_inspector.py", line 122, in <module>
    from tensorflow.python.platform import gfile
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/python/platform/gfile.py", line 22, in <module>
    from tensorflow.python.lib.io.file_io import copy as Copy
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/python/lib/io/file_io.py", line 27, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
    _pywrap_tensorflow = swig_import_helper()
  File "/opt/anaconda3/lib/python3.6/site-packages/tensorboard/tensorboard.runfiles/org_tensorflow/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/opt/anaconda3/lib/python3.6/imp.py", line 342, in load_dynamic
    return _load(spec)
SystemError: initialization of _pywrap_tensorflow raised unreported exception

Which version of numpy is used in the pypi package for tensorboard?

Seems we have some problems with Python3.6, there're some issues about this. Did you try another Python version?

Unfortunately, this anaconda is my only option as I'm on a department cluster. I also tried building from source, but since they have already changed the tensorflow repo that does not work either. Do you know which is the last working tag/commit from tensorflow for which I can build tensorboard with?

@botev I have fixed the backend and the install script. https://github.com/dmlc/tensorboard/blob/master/installer.sh#L9

Hmm, is there anything specific that I need to change during installation. When I run the build script after bazel I get:
Requirement 'python/dist/*.whl' looks like a filename, but the file does not exist

and there does not seem to be an wheels in python/dist

Could you tell me how to reproduce this? I will look into this.

So I just cloned the repo. Than did this and the output is:
$chmod +x installer.sh
$./installer.sh
protoc tensorboard/src/*.proto --python_out=python/
Cloning into 'tensorflow'...
remote: Counting objects: 225876, done.
remote: Total 225876 (delta 0), reused 0 (delta 0), pack-reused 225876
Receiving objects: 100% (225876/225876), 117.86 MiB | 1.84 MiB/s, done.
Resolving deltas: 100% (174956/174956), done.
Switched to a new branch 'v1.0.0-rc1'
Please specify the location of python. [Default is /opt/anaconda3/bin/python]:
Please specify optimization flags to use during compilation [Default is -march=native]:
Do you wish to use jemalloc as the malloc implementation? (Linux only) [Y/n]
jemalloc enabled on Linux
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with Hadoop File System support? [y/N]
No Hadoop File System support will be enabled for TensorFlow
Do you wish to build TensorFlow with the XLA just-in-time compiler (experimental)? [y/N]
No XLA support will be enabled for TensorFlow
Found possible Python library paths:
  /home/alex/work/python/data-sets
  /home/alex/work/python/Lasagne
  /home/alex/work/python/LaplaceNets
  /home/alex/work/python/Theano
  /opt/anaconda3/lib/python3.6/site-packages
Please input the desired Python library path to use.  Default is [/home/alex/work/python/data-sets]
/opt/anaconda3/lib/python3.6/site-packages
Do you wish to build TensorFlow with OpenCL support? [y/N]
No OpenCL support will be enabled for TensorFlow
Do you wish to build TensorFlow with CUDA support? [y/N]
No CUDA support will be enabled for TensorFlow
Configuration finished
... BAZEL BUILDS ....
Requirement 'python/dist/*.whl' looks like a filename, but the file does not exist
*.whl is not a valid wheel filename.
Now you can remove tensorflow with rm -rf tensorflow

Where are the wheels suppose to be at the end of the bazel build?

Did the bazel build success? Let me try this in my env.

I think so because tensorboard/python/tensorboard/tensorboard is a working tensorboard which I can run. I guess there something fishy going with the wheels. If you don't get it its fine, since I can just modify the source and use that.

I failed to reproduce this in Python2.7, let me try in Python3.6 later.

Sure if you need some more info let me know, just note that my python 3.6 is actually under anaconda. That should not matter, but I have experienced cases where it does.


Will it be faster to train a model on GPU rather than a CPU? how can I make it support OpenCL?

I'm trying to train some model, it takes many many hours to generate one. Actually I haven't got any of it after running scripts a whole day. Is it better to train it on a GPU?
I have  a AMD R2 920, but no NVIDIA, how to make this script runs on my AMD GPU?
Thank you.

Yes, it's better to use GPU. Only NVIDIA GPU's supported, for now, but as a hack  try to change this line
https://github.com/DmitryUlyanov/texture_nets/blob/master/train.lua#L66
to dtype = 'torch.ClTensor', replace all require 'cutorch'  with require 'cltorch' and remove all cudnn related stuff. I do not have AMD GPU so I cannot test it and push a commit myself.

Thank you but never mind, I bought a nvidia GTX1080 yestoday, it's working good, trying to get good results.
I will try it if place the AMD GPU back.


about the frame rate

Hi, cshizhe
I find that the number_of_feature/video_duration of videos are different, can you tell me the temporal interval of visual features?
Thanks

I want to know how you extract I3D features from a video. This I3D feature only uses RGB or flow and RGB？ Can you provide code to extract features?  Thanks!

For the resnet features, we extract the feature every 32 frames (for TGIF dataset, 8 frames).
For the I3D feature, we use the provided features from the VATEX team.

@cshizhe hello shizhe ! thanks for share the wonderful work and the code you write is like poem :)  I found that the vide clip in video0 is from 137.72 to 149.44 which means is about 12s duration  and 280 frames. But I find the corresponding feature shape in trn_ft.hdf5 is 19*2048. if I downsample as you say(every 32 frames) it should be 8 frame. It confused me a lot. Looking forward to your reply, thx.

@cshizhe hello shizhe ! thanks for share the wonderful work and the code you write is like poem :) I found that the vide clip in video0 is from 137.72 to 149.44 which means is about 12s duration and 280 frames. But I find the corresponding feature shape in trn_ft.hdf5 is 19*2048. if I downsample as you say(every 32 frames) it should be 8 frame. It confused me a lot. Looking forward to your reply, thx.

Sorry about my mistake in previous reply. The resnet feature is extracted every 16 frames for MSRVTT dataset. For example, the video0 contains 300 frames, and the extracted feature size is 19 (18.75 = 300/16).


Adjust dump format

We need to agree on dump format and change it accordingly before doing release.
Check conversation here:
https://gist.github.com/yifanlu/d546e687f751f951b1109ffc8dd8d903

I have updated the format. however flags and checksum still have to be handled.

sha256 is now calculated using native vita functions

trimmed flag is handled in df00aec755a15e553e46d0071ef5f7593dbc88ac


add empty player boxes to lobby

important: to make it clear how much players fit in a lobby (only mentioned in rules)

b595f00d68eb537940672e36b55d7fab39e257d8


Editor serializer

P tag introduced for each component of verse type annotation.
test case introduced for editor serializer.

P tag introduced for each component of verse type annotation.
test case introduced for editor serializer.


randomise within the timeline

Hi, I have been trying to draft the script for a relatively long online study. It is divided into different sections (see below). Different parts have different structures. What I want to do is for part 4,5,6, they can be randomised. But part_5_intro+part_5_main, part_6_intro+part_6_main always go together. This is because "intro" is the introduction before the "main" and there is some example questions before "main". anyone have an idea how to randomise 4,5,6.?
 timeline.push(part_1);
 timeline.push(part_2);
 timeline.push(part_3);

 timeline.push(part_4);
 timeline.push(part_5_intro);
 timeline.push(part_5_main);
 timeline.push(part_6_intro);
 timeline.push(part_6_main);

I have tried to combine part 5 into single set to ensure they can be presented together. e.g. var part5_set = [part_5_intro, part_5_main],  var part6_set = [part_6_intro, part_6_main], but this will leads to the problem: TimelineNode.ts:102 Trial level node is missing the "type" parameter. I understand that we need to have type parameter for each trial, but still not clear how to realise the shuffle in this case.
  // Prepare all questionnaires here (the ones to be shuffled)
   var all_questlist = [part4, part5_set, part6_set];

   // Shuffle all of questionnaires
   var shuffled_all_questlist = jsPsych.randomization.shuffle(all_questlist);

   // Initialize the processed list
   var processedQuestList = [];

    // Push the processed questionnaires into the timeline
    shuffled_all_questlist.forEach(quest => {
     timeline.push(quest);
   });

Turns out it is better to flat the array so the arrays can be pushed one by one. it will not run into the type error
var processedQuestList = [];
shuffled_all_questlist.forEach(quest => {
  if (Array.isArray(quest)) { // Check if the item is an array (i.e., part5_set or part6_set)
    // If it is, push each item in the array individually
    quest.forEach(part => timeline.push(part));
  } else {
    // If it's not an array, push it directly
    timeline.push(quest);
  }
});


DRILL-4250: Fix directory-based partition pruning when directory has …

…both subdirectory and file.
@amansinha100 , could you please review this small patch for directory-based partition pruning? Thanks!

+1 .


pod-tectonic-identity with error (console login and kubectl fails)

Tectonic Version
CURRENT VERSION: 1.4.7-tectonic.1
CHANNEL: Alpha
STRATEGY: Automatic
Kubernetes: 1.4.7+coreos.0
Tectonic: 1.4.7-tectonic.1
Environment
AWS
Expected Behavior
Tectonic Console and Kubectl keep working.
Actual Behavior
At some point, Console returns 503 when trying to log in and kubectl show the message below for any command:
kubectl cluster-info
error: error fetching provider config: invalid character '<' looking for beginning of value

I also noted that if it is already logged in, the console still works, and only kubectl shows the message above.
Reproduction Steps
This problem happens sporadically.
If the problem happens and i already logged in console, i see the pod tectonic-identity... with status ContainerIsNotReady,
See images bellow.

If delete the pod, other is created but in few seconds the error happens again.
Other Information
When the error happens and i'm not logged in console, i can't manage the cluster, but the access, return after many hours. Without any intervention.

The error is propagated for the others components in cluster.

@pagottoo identity will start failing health checks if the API server is unavailable. I think the error is propagating in the other direction since identity can't take down your control plane. Are you using the experimental operators and have updates on? Can you kubectl describe some of the failing containers?

Hey @pagottoo - were you able to resolve this issue? We recently released Tectonic v1.6.2-tectonic.1 with a ton of improvements - would recommend you check it out!

I had this errors with 1.6.2 ? Filed a bug in https://github.com/coreos/tectonic-installer/issues/1199
Any idea where to start looking?


do a todo: cleanup pass before 1.0 release

new TODO badge thing doesn't recognize any of my todo: comments, but does pick up a bunch of old ones from shiny...
anyway, can just find and remove / fix / file issues

also vet, goreport, spelling, etc.

done -- still more naggy lint stuff but nothing critical


Anti-monopolist ruleset

I got an email from Tom Fotheringham and he requested I add an issue here to bring it to your attention. I did mention 99% of the credit goes to you @intrepidcoder. It's an interesting concept and the guy is pretty cool.

Hello,

I just discovered the Monipoly game which you and Daniel Moyer wrote.  I have some ideas on how this could be made into a significant contribution to the field of political economics, specifically to the ideas of Henry George.  As you probably know Monopoly is based on the Landlord's Game patented in 1904 by Elizabeth Magie to educate users about Georgism.  There were two sets of rules, one monopolist which results in only one winner, the other anti- monopolist which results in all players winning (not losing).

With slight changes to the game you have written I believe you could achieve Magie's goal by allowing users to adjust the rules and have it played by all AI players letting the game run automatically for thousands of moves in seconds. The object would be to find the smallest changes to the standard rules that would result in the highest number of iterations without anyone losing.
The point of this would be to illustrate how our economic system could be adjusted to basically eliminate poverty.
If you're interested I'd love to share more.
I'm a retired programmer 75 years old. I could maybe do it myself but I'd rather share the project with others.
Let me know if you or anyone you know might be interested.

Tom Fotheringham
Ottawa, Canada

Seems like an idea for a mini research project. I don't know anyone who would be interested, but of course you or anyone else is welcome to fork the repo and try it. Although even with the existing rules, my experience is that when all the properties have been bought, no one player owns a single color group, and all players refuse trades that would give another player a full color group, is it very difficult to force a bankruptcy, so I could see the game lasting indefinitely in that scenario. I'm going to close since this isn't actually a bug in the project.


fix(plugin): change lifecycle event name of commands to command sin…

…ce it is exposed to users
fix #79

It looks good to me, thanks very much.

thanks to you!


Volume CR must be created in the same namespace as PVC/PV

@AnastasiaShemyakinskaya please add detailed description why we need these changes...

@AnastasiaShemyakinskaya please add detailed description why we need these changes...


Move check batch errors to builder

What have you changed? (mandatory)
In future executors, we will have multiple exit points in check_supported function, so it would be better to produce the error message outside, i.e. in builder, to reduce code duplication.
What are the type of the changes? (mandatory)

Engineering (engineering change which doesn't change any feature or fix any issue)

How has this PR been tested? (mandatory)
No need.

/run-integration-tests

/rebuild


Sample with local system interaction of a bot

Hello,
I would like to see an example where a BOT performs  local system tasks maybe as simpel as opening notepad
and / or get data from a local ( company ) resource as response.
The problem is that I work though the examples and only  see web related tasks and public resources as answer
Describe the solution you'd like
What I would like to see is a clear example on how a BOT client can interact with the system of the user
like opening a third party software , like in my case DocuWare of Ricoh as a response  and a  answer with companies data
what is the telephone number of customer X   ( and then fetch this from a CRM table  of a database  in the company)
Describe alternatives you've considered
If i could hook into the answer stream with a .Net client app I guess that could also solve my problem.
Additional context
We want to build a bot to help our customer service employees , so customer calls and the BOT would help the employee to get the needed information fast and easy , a few of the tasks it should do is ..
opening  a third party software ( Docuware from Ricoh ) with parameters that the QNA has as a result ( zipcode , customer number etc etc )   , but this would thus be on a companies client computer on a intranet .
If you have any follow-up questions please ask , if you find this case 'interesting' and want to help me and point me in the right direction , I would be happy to share so others can profit from the experience .

Thank you for the feature request but we are not considering this at this time. If you would like assistance with this specific scenario, please try Stack Overflow.


[NNPA] Document for supported ops for NNPA

This PR adds documentation about supported ops for NNPA, automatically generated by the same script as for the CPU.
The documentation for the CPU is also updated by this script to reflect latest update.
CPU is used instead of NNPA when onnx ops don't meet these limitations. There are another limitation in HW written in here . They are enough large for practical model, but they are written in top paragraph using newly added label ==ADDITIONAL_PARAGRAPH==

@tungld @AlexandreEichenberger Ready for review again. Thanks!

@AlexandreEichenberger I updated utils/documentOps.py to add additional paragraph above the table . If we write ==ADDITIONAL_PARAGRAPH== in parsed file, the sentence is inserted there. Could you review it?

Jenkins Linux amd64 Build #6523 [push] [NNPA] Document for supp... started at 20:00

Jenkins Linux ppc64le Build #5635 [push] [NNPA] Document for supp... started at 21:02

Jenkins Linux s390x Build #6539 [push] [NNPA] Document for supp... started at 21:00

Jenkins Linux amd64 Build #6523 [push] [NNPA] Document for supp... passed after 1 hr 8 min

Jenkins Linux s390x Build #6539 [push] [NNPA] Document for supp... passed after 1 hr 30 min

Jenkins Linux ppc64le Build #5635 [push] [NNPA] Document for supp... passed after 1 hr 47 min


Support building elastic stack in other regions

I wanted to standup my own pipeline for elastic-ci-stack-for-aws, and to do that we needed to:

build in regions other than us-east-1
build using buckets other than Buildkite's

To do this:

added or extended support for specifying parameters as pipeline environment variables
make buildkite-ami.json a generated artefact

Thanks @mikeknox! Mind rebasing this onto master so that it's a clean merge?

Unfortunately I think the scope of this change is probably going to be too big to accept without having our test suites run against it.
@mikeknox would you mind if I copy-paste some bits of this change into smaller branches?

I included a lot of the region agnostic parts of this PR in https://github.com/buildkite/elastic-ci-stack-for-aws/pull/401. Going to close this one out for now.


Fix footer copyright align

Before:

After:

Thanks you @tdtgit for taking the time to fix errors like this one.


zpay32: check route+hop hints while decoding

This commit checks that there are not greater than 20 route hints while decoding or more than 20 hop hints for a single route hint. This mitigates a potential DoS vector where an attacker could craft a very large bech32 invoice string containing an absurd amount of route and/or hop hints. If sent to an application that processes payment requests, this would allocate a burdensome amount of memory due to the public key parsing for each route/hop hint.
For a 1.7MB payment request, this yielded about 38MB in allocations from just parsing public keys (bech32VerifyChecksum allocated another 25MB):
   45.51MB  7.31% 92.07%    45.51MB  7.31%  math/big.nat.make
   25.50MB  4.09% 96.16%    25.50MB  4.09%  github.com/lightningnetwork/lnd/zpay32.bech32VerifyChecksum
       1MB  0.16% 96.32%    39.50MB  6.34%  github.com/lightningnetwork/lnd/zpay32.parseRouteHint
       1MB  0.16% 96.48%    33.50MB  5.38%  github.com/btcsuite/btcd/btcec.decompressPoint
    0.50MB  0.08% 96.56%     7.50MB  1.20%  crypto/elliptic.(*CurveParams).doubleJacobian
    0.50MB  0.08% 96.64%       38MB  6.10%  github.com/btcsuite/btcd/btcec.ParsePubKey
         0     0% 96.64%       12MB  1.93%  crypto/ecdsa.Verify
         0     0% 96.64%        8MB  1.28%  crypto/elliptic.(*CurveParams).ScalarBaseMult
         0     0% 96.64%       12MB  1.93%  crypto/elliptic.(*CurveParams).ScalarMult

With this change, memory usage will be far lower as decoding will exit early with an error if too many route/hop hints are found.
EDIT:
With https://github.com/btcsuite/btcd/pull/1429, the memory allocation from parsing pubkeys occupies about ~6.5MB which is significant improvement.    Another fix would be to limit the size of invoices before attempting any decoding.  In conjunction with the commits in this PR, it should limit any potential sources of memory blow-up.

I'd also like to include a length limit on the invoices that'll be checked before starting the decoding process.  Ideas @wpaulino ?
I'm by no means in expert in QR codes or anything, but from here https://en.wikipedia.org/wiki/QR_code#Storage it seems that len(invoice) < 7089 for it to fit into a QR code.  I think this might even be implementation depending on the app used.  However, if each field can have a maximum of 1024 bytes and there can be 20 route hints, this would give 20480 bytes, not counting the other data in the invoice.  With the current code, it's legal for an invoice to not fit into a QR code which I think was one of the points of bolt 11?
Another reason to limit the length besides the QR code argument is that you can also make invoices huge by including unknown fields that'll be skipped over while decoding.  So limiting route hints doesn't solve the memory blow up issues entirely.


How to perform the Geo-restore if the source region/sql server not available

Hello,
The document states:
"You can request a geo-restore even if the database or the entire region is inaccessible due to an outage."
However, if the original region\source SQL server is unavailable, the source database GRS copy is unavailable in the Azure portal drop-down and not available via PowerShell command.
Should this be done via a separate support ticket? How to test the geo restore in the case of a region outage?
Thanks,
Vlad

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1357a5f7-2e8e-9aab-5bc9-3a61f1a07f4f
Version Independent ID: 47497fe4-d8ab-87fc-3ca0-b46bbddb25f8
Content: Restore a database from a backup - Azure SQL Database
Content Source: azure-sql/database/recovery-using-backups.md
Service: sql-database
Sub-service: backup-restore
GitHub Login: @SudhirRaparla
Microsoft Alias: nvraparl

@vlad-kozlov Thank you for the feedback. We are actively investigating and will get back to you soon.

@SudhirRaparla Can you help us with this ask please?

Hi @vlad-kozlov - Can you please confirm what is the Backup Storage Redundancy set to for the source database?

@SudhirRaparla , it's Geo-redundant backup storage for the source database.

@vlad-kozlov - Sorry for the delayed response. If the Backup Storage Redundancy is set to Geo-redundant, then the backup will be automatically replicated to paired geo region. You can use Geo-Restore capability which is a self-serve capability and doesn't require any support ticket. Way to do that via Portal is:

Create Database
Choose name and other mandatory fields in Basics tab
Go to Additional Settings tab and for Use Existing data option select Backup. In the dropdown list you should see the database you want to restore even if the containing logical server is down.

If you do not see this happening, it is a bug. Can you please raise a support ticket. Please let me know if you have any questions.

Hi @vlad-kozlov , thanks for contacting Microsoft! As it seems Sudhir was able to address your question, I'll go ahead and close out this Git Issue but please feel free to comment should you need anything further.
Thanks again for reaching out and I hope you have a wonderful rest of your day!
Masha from the SQL Docs team
#please-close


Invalid datatype returned by tuya sdk for device requests - breaks Home Assistant Integration

The Tuya SDK returns strings instead of floats for various values retrieved from devices, although the API contract states that it will return a float.
This breaks integrations with lights, fans and covers on Home Assistant as covered in various issues, such as:

https://github.com/home-assistant/core/issues/114100
https://github.com/home-assistant/core/issues/83670
https://github.com/home-assistant/core/issues/111839

As this is an issue with the Tuya API/SDK returning an invalid datatype, it is unlikely to be addressed within Home Assistant (see https://github.com/home-assistant/core/pull/121731)
This unfortunately leaves many Tuya devices fundamentally broken in HASS and possibly many other projects.

Tuya has fixed the API in the backend and returns the desired data type. This issue should have been fixed.


ERROR stack traces are hard to read

This seems to be one bad side-effect of the code-changes made by @aurium when incorporating markdown rendering in the text terminal with the rich lib.
I'd like to preserve line-breaks in stack traces to facilitate reading them.

The bug in the screenshot above was fixed, but the issue with hard-to-read stack-traces remains. In order to fix this issue, we'll have to induce a stack trace (perhaps by reintroducing the bug from the screenshot).
To do so, replace this:
@condition
def metadata_file(family_directory,
                  metadata_pb=None):

with this:
@condition
def metadata_file(family_directory=None,
                  metadata_pb=None):

on Lib/fontbakery/profiles/googlefonts_conditions.py, but only on your local copy. Do not include this on actual commits.


Correct Example Inconsistencies

The examples had some inconsistencies which resulted in invalid MD (two empty lines in a row).

This change is

Codecov Report

Merging #197 into dev will not change coverage.
The diff coverage is n/a.

@@         Coverage Diff         @@
##            dev   #197   +/-   ##
===================================
  Coverage    95%    95%
===================================
  Files        15     15
  Lines      1113   1113
===================================
  Hits       1068   1068
  Misses       45     45

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 64f0d27...55cfd56. Read the comment docs.

This was an easy review :wink:

Reviewed 21 of 21 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Update structure.sql

Change table and collate encoding to utf16 - for correct save and display emdji (e.g. in Telegram game we using custom keyboard with text and emoji icon)

utf8mb4 already supports emoji.
If I'm not mistaken, changing to UTF16 won't add any advantage and will increase the database size for every value saved.
UTF8 is variable 1 to 4 bytes.
UTF16 is variable 2 or 4 bytes.
UTF32 is fixed 4 bytes.
See here: http://stackoverflow.com/a/496340
@jacklul @MBoretto Am I correct here or have I missed something?

That's the first time that I hear about this encoding, we switch to utf8mb4 some months ago to handle all the possible characters.  @aleksraiden which characters are missing?
First results found on google:
https://dev.mysql.com/doc/refman/5.5/en/charset-unicode-utf16.html

So? I don't see any advantage to move to this encoding, would like to close this. @noplanman @jacklul ?


[CARBONDATA-1309] During Global Sort the DataWriterProcessorStepImpl not getting shutdown even when datalaod is finished.

Global Sort is not closing the DataWriterProcessorStepImpl because of this the thread started to print the number of records processed by the step at the interval of 10 second is not getting shutdown.

Can one of the admins verify this patch?

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder/3150/

Can one of the admins verify this patch?

LGTM


Model Builder Error "Could not find input column 'ImageSource' Parameter name: inputSchema" when adding WebAPI project to solution.

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog): <IP_ADDRESS>2802
Microsoft Visual Studio Enterprise 2019: 16.11.5

Describe the bug

On which step of the process did you run into an issue: Object detection>Consume page: Add WebAPI project to solution

To Reproduce
Steps to reproduce the behavior:

Select Create a new project from the Visual Studio 2019 start window;
Choose the C# Console App (.NET Core) project template with .Net 5.0;
Add model builder by right click on the project;
Select Object detection to complete training;
Navigate to Consume page, add WebAPI to solution, prompt Model Builder Error "Could not find input column 'ImageSource' Parameter name: inputSchema".

Expected behavior
No Error when adding projects to solution.
Screenshots

Error message
at Microsoft.ML.Data.OneToOneTransformerBase.CheckInput(DataViewSchema inputSchema, Int32 col, Int32& srcCol)
at Microsoft.ML.Data.OneToOneTransformerBase.OneToOneMapperBase..ctor(IHost host, OneToOneTransformerBase parent, DataViewSchema inputSchema)
at Microsoft.ML.Transforms.Image.ImageResizingTransformer.Mapper..ctor(ImageResizingTransformer parent, DataViewSchema inputSchema)
at Microsoft.ML.Transforms.Image.ImageResizingTransformer.MakeRowMapper(DataViewSchema schema)
at Microsoft.ML.Data.RowToRowTransformerBase.GetOutputSchema(DataViewSchema inputSchema)
at Microsoft.ML.Data.TransformerChain1.GetOutputSchema(DataViewSchema inputSchema)    at Microsoft.ML.ModelBuilder.AutoMLService.ServiceFactory.CodeGeneratorService.CreateWebApiProjectCodesAsync(TrainingConfiguration configuration, Dictionary2 sampleData, String sampleProjectName, String modelpath, String modelFileName, String className, ColumnDataKind labelColumnDataKind, String targetFramework, String[] labels, String pipeline, Boolean isAzureArtifact, Boolean isDEV17, CancellationToken ct) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/ServiceFactory/CodeGeneratorService.cs:line 150   at StreamJsonRpc.JsonRpc.d__1391.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.ML.ModelBuilder.ToolWindows.ConsumeViewModel.d__77.MoveNext()

This issue happens on Azure environment ( Image classification and Object detection),  and also see the error when predicting the model on Evaluate page of Image classification>Azure.

The code gen is broken for all image scenarios, I'll fix that.

@LittleLittleCloud this error still happens when trying the model on Evaluate page (Image classification + Azure).

ML.Net Model Builder: <IP_ADDRESS>5801 ( Latest Main)

Not happens when adding WebAPI project to solution.

not repro this issue on latest main: <IP_ADDRESS>6701


List of passes gets printed twice (one is empty)

For a while now, at the startup of the translation (config building), the list of possible passes gets printed twice. One seems to be correct, the other one is always empty
09:06:25,770 INFO  TranslationConfiguration$Builder Passes before enforcing order: [TypeHierarchyResolver, ImportResolver, SymbolResolver, DFGPass, DynamicInvokeResolver, EvaluationOrderGraphPass, TypeResolver, ControlFlowSensitiveDFGPass, FilenameMapper, DynamicInvokeResolver, CXXExtraPass]
09:06:25,784 INFO  TranslationConfiguration$Builder The following mermaid graph represents the pass dependencies:

flowchart TD;
   CXXExtraPass-->EvaluationOrderGraphPass;
   TypeResolver-->TypeHierarchyResolver;
   TypeHierarchyResolver-->ImportResolver;
   TypeResolver-->SymbolResolver;
   TypeHierarchyResolver-->SymbolResolver;
   EvaluationOrderGraphPass-->SymbolResolver;
   SymbolResolver-->DFGPass;
   SymbolResolver-->DynamicInvokeResolver;
   DFGPass-->DynamicInvokeResolver;
   EvaluationOrderGraphPass-->ControlFlowSensitiveDFGPass;
   DFGPass-->ControlFlowSensitiveDFGPass;
   TypeResolver-->CXXExtraPass;

09:06:25,794 DEBUG TranslationConfiguration$Builder Working list after initial scan: [PassWithDependencies[pass=TypeHierarchyResolver,softDependencies=[],hardDependencies=[TypeResolver]], PassWithDependencies[pass=ImportResolver,softDependencies=[],hardDependencies=[TypeHierarchyResolver]], PassWithDependencies[pass=SymbolResolver,softDependencies=[],hardDependencies=[TypeResolver, TypeHierarchyResolver, EvaluationOrderGraphPass]], PassWithDependencies[pass=DFGPass,softDependencies=[],hardDependencies=[SymbolResolver]], PassWithDependencies[pass=DynamicInvokeResolver,softDependencies=[],hardDependencies=[SymbolResolver, DFGPass]], PassWithDependencies[pass=EvaluationOrderGraphPass,softDependencies=[CXXExtraPass],hardDependencies=[]], PassWithDependencies[pass=TypeResolver,softDependencies=[],hardDependencies=[]], PassWithDependencies[pass=ControlFlowSensitiveDFGPass,softDependencies=[],hardDependencies=[EvaluationOrderGraphPass, DFGPass]], PassWithDependencies[pass=FilenameMapper,softDependencies=[],hardDependencies=[]], PassWithDependencies[pass=CXXExtraPass,softDependencies=[],hardDependencies=[TypeResolver]]]
09:06:25,795 DEBUG TranslationConfiguration$Builder Working list after adding missing dependencies: [PassWithDependencies[pass=TypeHierarchyResolver,softDependencies=[],hardDependencies=[TypeResolver]], PassWithDependencies[pass=ImportResolver,softDependencies=[],hardDependencies=[TypeHierarchyResolver]], PassWithDependencies[pass=SymbolResolver,softDependencies=[],hardDependencies=[TypeResolver, TypeHierarchyResolver, EvaluationOrderGraphPass]], PassWithDependencies[pass=DFGPass,softDependencies=[],hardDependencies=[SymbolResolver]], PassWithDependencies[pass=DynamicInvokeResolver,softDependencies=[],hardDependencies=[SymbolResolver, DFGPass]], PassWithDependencies[pass=EvaluationOrderGraphPass,softDependencies=[CXXExtraPass],hardDependencies=[]], PassWithDependencies[pass=TypeResolver,softDependencies=[],hardDependencies=[]], PassWithDependencies[pass=ControlFlowSensitiveDFGPass,softDependencies=[],hardDependencies=[EvaluationOrderGraphPass, DFGPass]], PassWithDependencies[pass=FilenameMapper,softDependencies=[],hardDependencies=[]], PassWithDependencies[pass=CXXExtraPass,softDependencies=[],hardDependencies=[TypeResolver]]]
09:06:25,795 INFO  TranslationConfiguration$Builder Passes after enforcing order: [[TypeResolver], [TypeHierarchyResolver, CXXExtraPass], [ImportResolver, EvaluationOrderGraphPass], [SymbolResolver], [DFGPass], [DynamicInvokeResolver, ControlFlowSensitiveDFGPass], [FilenameMapper]]
09:06:25,799 INFO  TranslationConfiguration$Builder Passes before enforcing order: []
09:06:25,799 INFO  TranslationConfiguration$Builder The following mermaid graph represents the pass dependencies:

flowchart TD;

09:06:25,799 DEBUG TranslationConfiguration$Builder Working list after initial scan: []
09:06:25,800 DEBUG TranslationConfiguration$Builder Working list after adding missing dependencies: []
09:06:25,800 INFO  TranslationConfiguration$Builder Passes after enforcing order: []

Also the worklist output (although in debug) is quite bloated, maybe we can do a more concise output here with some toString() implementations of PassWithDependencies

@oxisto what bothers you about the output of PassWithDependencies I guess it won't get much more concise if we want to keep the difference between soft and hard dependencies of all the passes in the output. I'd skip empty lists though


Clarify ExternalName binary emit usage

So it looks like ExternalName is used in binemit and other places.  E.g., wasmstandalone uses it in function name emission:
https://github.com/sunfishcode/wasmstandalone/blob/04660ae8cc419cc9becdc675788c5393b78f872b/lib/runtime/src/lib.rs#L36
The problem with this is that the symbol name is typically prefixed with a %; this is fine for internal usage, but for binaries, this will be very problematic for tools like gdb which won't allow disassembly of a function with a % prefixed, e.g.:
(gdb) disass 0x0
Dump of assembler code for function %wasm_0x0:
   0x0000000000000000 <+0>:	rex mov $0x0,%eax
   0x0000000000000006 <+6>:	rex mov $0x0,%eax
   0x000000000000000c <+12>:	callq  0x11 <%wasm_0x0+17>
   0x0000000000000011 <+17>:	retq
End of assembler dump.
(gdb) disass %wasm_0x0
A syntax error in expression, near `%wasm_0x0'.
(gdb) disass \%wasm_0x0
Invalid character '\' in expression.

Should another function be used in this case, or should the client take care to emit different function names?

The '%' prefix is added by fmt/to_string/etc. when printing, which is meant for Cretonne's text format. If you want the raw bytes, there's an as_ref() which returns a &[u8].

Hmm. So specifically, for, e.g. filetests/call.wat:
(module
  (func $main (local i32)
    (set_local 0 (i32.const 0))
    (drop (call $inc))
  )
  (func $inc (result i32)
    (i32.const 1)
  )
  (start $main)
)

I was hoping to see $main and $inc
How hard would that be to return for the function name in current wasmstandalone?

In wasm, function names aren't semantically meaningful, which is why I haven't wired them up yet. The names we'd really want would be the export names, such as if you added (export "public_name_for_inc" (func $inc)). It wouldn't be complicated to take the names from the exports and use them for function symbols.

Also, I just realized that the '%' names were coming from wasm2obj, so I've now pushed a patch to fix that.

Ok I think its safe to close this, since iiuc:

export names (if specified in wasm file) are the external names wasm2obj needs to omit
otherwise names/functions are local (still would be nice to use the local name for debugging, etc.)
the % was minor bug in wasm2obj and is fixed now


Authenticator info logging may be too verbose

Expected behaviour

Authenticator only logs events that are useful operationally at INFO level

Actual behaviour

Authenticator logging unauthenticated users (is this useful?)

Steps to reproduce the behaviour

Check server logs
see "Session does not have an authenticated token" logging every few seconds

Technical information
Logging line is here: /app/authentication/authenticator.py:37
Screenshot

For discussion, maybe this is useful afterall

This log message is output when the root '/' url is visited. This looks like it's because flask-login is initialised in app/init.py so happens for all requests regardless whether the user should be logged in or not.

Let's leave this issue open to review the flask-login usage for various routes then.

Is this still a problem, since we're now using structured logging as well?

Logs less now


[BUG] Failed GitHub Action Workflow

What steps does it take to reproduce the issue?

When does this issue occur?
When creating a ticket #2107 and #2108 I got a failed Github action run

See: https://github.com/Islandora/documentation/actions/runs/2341619959 and https://github.com/Islandora/documentation/actions/runs/2341629044

Which page does it occur on?
When submitting issues

What happens?
Generated a failed job

To whom does it occur (anonymous visitor, editor, administrator)?
anyone

What did you expect to happen?
Run tasks, but not fail?

Which version of Islandora are you using?
N/A
Any related open or closed issues to this bug report?
N/A
Screenshots:
N/A

I believe @islandora-community was playing around with project boards and was trying out an action to move issues on a project board (https://github.com/peter-evans/create-or-update-project-card). It does not appear we have a project board right now, which (I'm guessing) is why it fails.

Possibly need to add a Token from the organziation to the workflow?
This was from this example: https://github.com/peter-evans/create-or-update-project-card#create-a-card-in-an-organization-or-user-project


Is there any possible to print the intermediate result of each layer?

Problem description:
I just run ResNet on the embedded device, but the output label is not correct. So I just wondering is there any way to see the input and output of each layer?  I use the Example object to setup the ResNet.

Unfortunately we don't have anything that will print automatically. You have to add the prints manually in each layer you are interested. Here's a couple of thing that should speed up the process:
First compile in debug mode, here are example:
scons os=linux arch=arm64-v8a Werror=1 mode=debug neon=1 opencl=1 examples=1 validation_tests=1 benchmark_tests=1 -j32
This will enable the print function of a tensor. CLTensors work a little different from Tensor (used by NEON). Since you didn't specify I will go with NEON which is a little simpler. The idea would be to print the output tensor of the kernels used by the network after the result has been computed. Here an example:
output_tensor->print(std::cout)
Checking which kernel are being used manually might be time consuming, so you can use one of the benchmark functionalities to check the used kernels for you. Here is the command:
./tests/benchmark_graph_resnet50 --instruments=SCHEDULER_TIMER_MS --example_args=--layout=NHWC,--target=NEON

...

  SchedulerTimer/block1/unit1/bottleneck_v1/Relu/NEActivationLayerKernel #16:    AVG=1.8680 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/add/NEArithmeticAdditionKernel #15:    AVG=4.1030 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/add/NEFillBorderKernel #14:    AVG=0.0030 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv1/BatchNorm/NEBatchNormalizationLayerKernel #5:    AVG=2.7400 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv1/convolution/NEGEMMAssemblyWrapperKernel/sgemm_nativeA_pretransposeB_16x4 #4:    AVG=5.4380 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/BatchNorm/NEBatchNormalizationLayerKernel #11:    AVG=9.8840 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/BatchNorm/NEBatchNormalizationLayerKernel #9:    AVG=2.8020 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/convolution/NEGEMMAssemblyWrapperKernel/sgemm_nativeA_pretransposeB_16x4 #7:    AVG=12.4590 ms

Let's take NEActivationLayerKernel as example, here where you should add you code:
https://github.com/ARM-software/ComputeLibrary/blob/master/src/core/NEON/kernels/NEActivationLayerKernel.cpp#L572
Which should become:
void NEActivationLayerKernel::run(const Window &window, const ThreadInfo &info)
{
    ARM_COMPUTE_UNUSED(info);
    ARM_COMPUTE_ERROR_ON_UNCONFIGURED_KERNEL(this);
    ARM_COMPUTE_ERROR_ON_INVALID_SUBWINDOW(IKernel::window(), window);
    ARM_COMPUTE_ERROR_ON(_func == nullptr);

    (this->*_func)(window);
    _output->print(std::cout);
}

Hope this helps
Thanks

Unfortunately we don't have anything that will print automatically. You have to add the prints manually in each layer you are interested. Here's a couple of thing that should speed up the process:
First compile in debug mode, here are example:
scons os=linux arch=arm64-v8a Werror=1 mode=debug neon=1 opencl=1 examples=1 validation_tests=1 benchmark_tests=1 -j32
This will enable the print function of a tensor. CLTensors work a little different from Tensor (used by NEON). Since you didn't specify I will go with NEON which is a little simpler. The idea would be to print the output tensor of the kernels used by the network after the result has been computed. Here an example:
output_tensor->print(std::cout)
Checking which kernel are being used manually might be time consuming, so you can use one of the benchmark functionalities to check the used kernels for you. Here is the command:
./tests/benchmark_graph_resnet50 --instruments=SCHEDULER_TIMER_MS --example_args=--layout=NHWC,--target=NEON

...

  SchedulerTimer/block1/unit1/bottleneck_v1/Relu/NEActivationLayerKernel #16:    AVG=1.8680 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/add/NEArithmeticAdditionKernel #15:    AVG=4.1030 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/add/NEFillBorderKernel #14:    AVG=0.0030 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv1/BatchNorm/NEBatchNormalizationLayerKernel #5:    AVG=2.7400 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv1/convolution/NEGEMMAssemblyWrapperKernel/sgemm_nativeA_pretransposeB_16x4 #4:    AVG=5.4380 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/BatchNorm/NEBatchNormalizationLayerKernel #11:    AVG=9.8840 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/BatchNorm/NEBatchNormalizationLayerKernel #9:    AVG=2.8020 ms
  SchedulerTimer/block1/unit1/bottleneck_v1/conv2/convolution/NEGEMMAssemblyWrapperKernel/sgemm_nativeA_pretransposeB_16x4 #7:    AVG=12.4590 ms

Let's take NEActivationLayerKernel as example, here where you should add you code:
https://github.com/ARM-software/ComputeLibrary/blob/master/src/core/NEON/kernels/NEActivationLayerKernel.cpp#L572
Which should become:
void NEActivationLayerKernel::run(const Window &window, const ThreadInfo &info)
{
    ARM_COMPUTE_UNUSED(info);
    ARM_COMPUTE_ERROR_ON_UNCONFIGURED_KERNEL(this);
    ARM_COMPUTE_ERROR_ON_INVALID_SUBWINDOW(IKernel::window(), window);
    ARM_COMPUTE_ERROR_ON(_func == nullptr);

    (this->*_func)(window);
    _output->print(std::cout);
}

Hope this helps
Thanks

Thank you very much for replying me. My target core is GPU, so it's OpenCL. I will try this method on CL layers. I just checked the GraphManager.cpp code. And got the GraphManager::exectue_graph(Graph &graph) function. It has a chain of layers called "it->second" which seems to hold tensors "input" and "output". Hope I can use the print function of these two tensors to get the intermediate results.

You cannot dump all the tensors, however you can modify the graph to add an output tensor to see what the tensor contains.
For example:
graph << Layer_0() << Layer_1() << Layer_2() << Layer_3() << OutputLayer(OutputAccessor());

If you want to check the output of Layer_1 just change the graph to:
graph << Layer_0() << Layer_1() << OutputLayer(OutputAccessor());

And print the content of the tensor in your OutputAccessor()

Hi, AnthonyARM
Many thanks to telling me this. Can you give me some details about the OutputAccessor() parameters. According to the examples, they are related to the number of the items of the label. I've no idea how to set up them.

You need to create an Accessor class which inherits from ITensorAccessor, a bit like this (Note: I haven't actually tried to compile this code):
class PrintAccessor final : public graph::ITensorAccessor
{
public:
    PrintAccessor()=default;
    /** Allows instances to move constructed */
    PrintAccessor(PrintAccessor &&) = default;

    // Inherited methods overriden:
    bool access_tensor(ITensor &tensor) override
    {
         tensor.print(std::cout);
         return true;
    }
};


Gp 3585.api shesmu plugin priority

Jira ticket:

[ x] Includes a change file
[ ] Updates developer documentation

The way you've already implemented the code, priority is a key-value pair submitted as part of the existing consumableResources component of a workflow run submission request (see here for that API description). I don't think there needs to be any additional API modifications, unless you're planning to change how you're implementing the feature?


Fixed incorrect type display of function arguments in GDScript editor tooltips

For function arguments without a defined type but with a default value set, the arguments type will be displayed as the type of the default value in the editor tooltip (should be Variant)
Fix #63017
Before this PR:

After:

Also note that for the current Master branch, the default value of the function parameter will not be displayed in the tooltip, so the picture shows the effect after merging #62934 and #62994

Needs rebase to fix CI due to a temporary godot-cpp issue.

Rebased to fix CI issue