sorry, too many clients already

Hello everyone!
We have decided to use the latest version of ent for the first time in our services. it is pretty good in many things but we got a weird bug.
in our code we use this to create the ent instance for interacting with database:
dsn := fmt.Sprintf("host=%s port=%s user=%s dbname=%s password=%s sslmode=disable", host, port, username, databaseName, password)

client, err := ent.Open("postgres", dsn)
if err != nil {
	log.Fatalf("Failed opening connection to postgres: %v", err)
}

if migrationErr := client.Schema.Create(context.Background()); migrationErr != nil {
	log.Fatalf("Failed creating schema resources: %v", migrationErr)
}

databaseWrapper := &DatabaseWrapper{
	Client: client,
	Logger: logger,
}

we use this wrapper every where in the code.
couple of days ago we decided to run a k6 script for load test. It was working completely fine before this. while testing we got an error which says:
sorry, too many clients already

Based on comment, ent.Open() is supposed to create a client and return it, but it seems that is not a real client and something else is happening under the hood, because we get too many clients error
Thank you!

sql.Open creates a sql.DB, which holds a pool of connections to the database. The error you're encountering is returned by PostgreSQL because there are too many idle or busy connections open to the database (you can read more about it here). You can manage the connection pooling settings in the database/sql package, like max idle/open connections, etc.
Closing as this issue is not something in Ent, but feel free to join our Discord and ask for help if you need.


Git submodules

From @Floppy on May 31, 2017 9:58
Probably a low priority niche use case, but a new "language" for updating git submodule versions would be incredibly useful (at least for me).
I have a number of Jekyll sites that include content from other repositories via submodules: https://github.com/SomethingNewUK/somethingnewuk.github.io/ is probably the most complex of those, with about 4 submodules in various subdirectories.
User story attempt:

As a website maintainer who does silly things with Jekyll
I would like to be able to add a "git submodule" language to a repository
So that I can get an automatic PR created when the submodule's repository has been updated

Copied from original issue: dependabot/feedback#4

From @hmarr on May 31, 2017 10:13
In theory, I'm 👍 - it'd be a nice feature to support, and the implementation shouldn't be too tricky.
We'll update this issue when we have a more concrete plan.

Just added support for these. @Floppy - if you run it on the repo you mentioned you'll get a resolution error, because the branch you specify for _data/popolo-data has been deleted. We could do better on the error message, but it'll do for now :)
🎉

From @Floppy on September 11, 2017 9:6
Not sure it's working, unless there's still something wrong on my repo. I've fixed the missing branch (thanks for spotting that!) so I think it should work now on https://github.com/SomethingNewUK/somethingnewuk.github.io.
However, while there are two submodules that need updating (_data/finances and _data/popolo_data) there, no PRs are appearing, whether on an automatic or manual bump.
Any thoughts?

 :D

From @Floppy on September 11, 2017 9:8
Ignore me! I just realised I need to add it as another language! I'll report back with results.

🎉
@deivid-rodriguez - I saw you got an error bumping submodules. At the moment it can't handle anything other than GitHub ones, but I'll hook it into the git internal transfer protocols this evening and get that fixed.

From @Floppy on September 11, 2017 9:8
BOOM 💥
It works - I love you guys :D


nodeos report errors when  a new  bp join into private network

42500ms thread-0   producer_plugin.cpp:569       block_production_loo ] Produced block<PHONE_NUMBER>dad8f4... #147 @ 2018-05-14T07:14:02.500 signed by eosio [trxs: 0, lib: 147, confirmed: 0]
842560ms thread-0   net_plugin.cpp:703            connection           ] accepted network connection
842693ms thread-0   block_log.cpp:206             read_block_by_num    ] 13 NSt8ios_base7failureE: basic_ios::clear
rethrow basic_ios::clear:
{"what":"basic_ios::clear"}
thread-0  block_log.cpp:206 read_block_by_num
842693ms thread-0   net_plugin.cpp:1042           enqueue_sync_block   ] write loop exception
843000ms thread-0   block_log.cpp:177             append               ] 10 assert_exception: Assert Exception
my->index_stream.tellp() == sizeof(uint64_t) * (b->block_num() - 1): Append to index file occuring at wrong position.
{"position":8,"expected":1168}
thread-0  block_log.cpp:167 append
843000ms thread-0   producer_plugin.cpp:545       block_production_loo ] Got exception while generating block:
10 assert_exception: Assert Exception
my->index_stream.tellp() == sizeof(uint64_t) * (b->block_num() - 1): Append to index file occuring at wrong position.
{"position":8,"expected":1168}
thread-0  block_log.cpp:167 append
rethrow
{}
thread-0  block_log.cpp:177 append
843000ms thread-0   producer_plugin.cpp:591       block_production_loo ] exception producing block
843500ms thread-0   block_log.cpp:177             append               ] 10 assert_exception: Assert Exception
my->index_stream.tellp() == sizeof(uint64_t) * (b->block_num() - 1): Append to index file occuring at wrong position.
{"position":8,"expected":1168}
thread-0  block_log.cpp:167 append
843500ms thread-0   producer_plugin.cpp:545       block_production_loo ] Got exception while generating block:
10 assert_exception: Assert Exception
my->index_stream.tellp() == sizeof(uint64_t) * (b->block_num() - 1): Append to index file occuring at wrong position.
{"position":8,"expected":1168}
thread-0  block_log.cpp:167 append
rethrow
{}
thread-0  block_log.cpp:177 append

Can you share your config and details steps?

We haven't received any update for this. If you still have the problem and wish to discuss further, please post it in https://eosio.stackexchange.com


Filter queryset by currently authenticated user

Hello @everyone,
I am new to Django and I recently came across this styleguide to develop Django projects and I instantly fell in love with it.
I am facing a slight issue regarding how to filter a queryset using the currently authenticated user stored in request.user and also create a reusable base filterset that filters by the current user.
I have looked this up and the solution that I have mostly seen online is to create a custom filter backend. But since this styleguide uses selectors to filter querysets, the above solution will not work.
Can someone please point me in the right direction? Thanks in advance.

Hello @dagasatvik10 :wave:
Can you provide an example about where you want to filter that queryset, based on request.user? So I can answer you properly.
A code snippet + more explanation would help us point you to the right dirrection.
Cheers!

Hi @RadoRado
Thanks for the response. Here is an example of how I want to filter the queryset-
# files/models.py
class File(BaseModel):
    file = models.FileField(upload_to=file_generate_upload_path, blank=True, null=True)

    original_file_name = models.TextField()

    file_name = models.CharField(max_length=255, unique=True)
    file_type = models.CharField(max_length=255)

    # As a specific behavior,
    # We might want to preserve files after the uploader has been deleted.
    # In case you want to delete the files too, use models.CASCADE & drop the null=True
    uploaded_by = models.ForeignKey(BaseUser, null=True, on_delete=models.SET_NULL)

    upload_finished_at = models.DateTimeField(blank=True, null=True)

    @property
    def is_valid(self):
        """
        We consider a file "valid" if the the datetime flag has value.
        """
        return bool(self.upload_finished_at)

    @property
    def url(self):
        if settings.FILE_UPLOAD_STORAGE == FileUploadStorage.S3:
            return self.file.url

        return f"{settings.APP_DOMAIN}{self.file.url}"

# files/apis.py
class FileListByCurrentUserApi(ApiAuthMixin, APIView):
    class Pagination(LimitOffsetPagination):
        default_limit = 1

    class FilterSerializer(serializers.Serializer):
        id = serializers.IntegerField(required=False)
        file_type = serializers.CharField(required=False)

    class OutputSerializer(serializers.ModelSerializer):
        class Meta:
            model = File
            fields = ("id", "file_name", "file_type", "url")

    def get(self, request):
        # Make sure the filters are valid, if passed
        filters_serializer = self.FilterSerializer(data=request.query_params)
        filters_serializer.is_valid(raise_exception=True)

       # pass current user to selector
        files = file_list(filters=filters_serializer.validated_data, user=request.user)

        return get_paginated_response(
            pagination_class=self.Pagination,
            serializer_class=self.OutputSerializer,
            queryset=files,
            request=request,
            view=self,
        )

# files/selectors.py
def file_list(*, filters=None, user=None) -> QuerySet[BaseUser]:
    filters = filters or {}

    qs = File.objects.all()

    if user is None:
        return FileFilter(filters, qs).qs

    # filter by the current user
    return FileFilter(filters, qs).qs.filter(user=user)

# files/filters.py
class FileFilter(django_filters.FilterSet):
    class Meta:
        model = File
        fields = ("id", "file_type")

But when I was looking online on how to filter by the current user, I came across the following multiple times-
1. Create a custom filter backend that filters queryset by the current user-
# core/filters.py
from django_filters import rest_framework as filters

class IsOwnerFilterBackend(filters.BaseFilterBackend):
    """
    Filter that only allows users to get their own objects
    """
    def filter_queryset(self, request, queryset, view):
        return queryset.filter(user=request.user)

2. Use the custom filter backend in the API you want to filter queryset by the current user-
# files/apis.py
class FileListByCurrentUserApi(generics.ListAPIView):
    serializer_class = FileSerializer
    queryset = File.objects.all()
    filter_backends = (IsOwnerFilterBackend,)
    filterset_fields = ("user",)

So I can use this custom filter backend in any API where I want to filter by the current user.
But something like this is not possible in the styleguide as it uses selectors. So is my method correct or is there some better way to filter by current user that can be reused by any API?

Aha, I got it.
There are several options that you can do here:
Give an already filtered queryset to the filters:
def file_list(*, filters=None, user=None) -> QuerySet[BaseUser]:
    filters = filters or {}

    qs = File.objects.all()

    if user is None:
        qs = qs.filter(user=user)

    return FileFilter(filters, qs).qs

Then, you'll have a limited queryset, to work on, for your filter.
The other thing you can do is to pass the user as part of the filters & have a special filter method for that:
class FileFilter(django_filters.FilterSet):
    user = CharFilter(method="filter_user")

    class Meta:
        model = File
        fields = ("id", "file_type")

    def filter_user(self, queryset, name, value):
        return queryset.filter(user_id=value)

And then just pass that:
def file_list(*, filters=None, user=None) -> QuerySet[BaseUser]:
    filters = filters or {}
    if user is not None:
        filters[user] = user.id

...

I'd personally go with the first approach - just filter the queryset, before giving it to the filter set.
And of course, I'm not a big fan of giving the request to the filter set, since the filter should operate on querysets & should not care about the request.
Hopefully, this will help you :+1:

Thanks for the clarification @RadoRado.


disable flag in some pages

Details
i used    <OgImage /> in my default layout.. is there a way for me to disable it on some pages because on some pages i want to explicitly set ogImage and then as i do change it using useServerSeoMeta twitter image src remains the og version.. is there i way around this or i have to put that component in all pages but those where i want to override it

Hi, in v3 you can use defineOgImage(false) to remove it, or just use defineOgImage({ url: '/your-image.png' }) to replace it with another image.


Simplify logging in using the keyboard

Is your feature request related to a problem?
When authentication is enabled, depending on the browser it is currently hard or not possible to log in using the keyboard alone. In particular, to log in through the keyboard one has to currently carry out the following actions on the login screen:

Press Tab to switch to the Username field
Enter the username
Press Tab to switch to the Password field
Enter the password
Press Tab to switch to the "Log In" button (works on Chrome but on Safari) and press Enter

What is the expected behavior?
To enable/simplify logging in through the keyboard, it would help to revise the login screen as follows:

[ ] When the login screen loads, set the focus automatically on the Username field
[ ] While inside the form, allow the Enter key to act as if the user pressed the Log In button

Additional context (if any):
To reproduce the issue, edit Label Sleuth's configuration file to enable authentication as described here and start the tool.

Suggested solution:

For item 1: use the autoFocus prop that can be passed to the Textfield components.
For item 2: pass a onKeyDown() function to the Box component that holds the login form. Such a function should check if the key that was pressed is Enter and if so, call the handleLogin() function. You can import the KeyboardKeysEnum from const.ts in order to not to hardcode the 'Enter' string. As this is a typescript file, the event that is passed to the function has to be typed. The event type should be React.KeyboardEvent so that you can access its key attribute.


Use of DataReaderFactory.register_service

I extended DataReaderFactory with:
class DataReaderFactory(FactoryInterface):
_reader_map = {
"xpt": XPTReader,
"json": DatasetJSONReader,
}
which indeed seems to associate the file suffix ".json" with the DatasetJSONReader (which works fine standalone).
But at this moment, I can only let the DatasetJSONReader work when I hardcode (which one shouldn't do ...):
def get_service(self, name: str = None, **kwargs) -> DataReaderInterface:
    """
    Get instance of service that matches searched implementation
    """
    print("get_service, name = ", name)
    **if name is None:
        name = "json"
    print("name changed into = ", name)**
    service_name = name or self._default_service_name

It looks as I also need to register DatasetJSONReader using "register_service", but I have no idea from where.
The only place where I can find this in test_data_reader_factory.py which is just for testing ...
As I understand it, the reader for XPT is never registered to DataReaderFactory as it is the default reader.
So, I wonder how to take care that also the DatasetJSONReader can be put to work.
Also still fighting with better understanding GitHub branching ...

This was resolved, correct code in https://github.com/JozefAerts/cdisc-rules-engine/tree/dev_Jozef_DatasetJSON.
Will do a pull request in the next few days.


Write unit tests for getFutureEvents

It's at src/utils/formatDate.js. You can run the tests with npm run test.

I can take this!

Awesome! Thanks @danmakenoise!

@agarrharr when working on this, I noticed getLastEndDate returns the earliest end date, not the latest. Is that intentional? It looks like the test descriptions for it don't match the expectations. Let me know what the truth is for that, and I can't make an issue if it's incorrect.

Actually, that's not intentional. Thanks for catching that.

Created PR! https://github.com/lifestonechurch/lifestonechurch.net/pull/496


Handle internal server errors while configuring secure boot

At least on some Dell machines, the Redfish SecureBoot resource is
unavailable during configuration, GET requests return HTTP 503.
Sushy does retry these, but not for long enough (the error message
suggests at least 30 seconds, which would be too much to just integrate
in Sushy). This change treats internal errors the same way as
mismatching "enabled" value, i.e. just waits.
Change-Id: I676f48de6b6195a69ea76b4e8b45a034220db2fa
(cherry picked from commit a6e3a7f50cd8594520cf80fa4ef0a07646221809)

/lgtm


add progress bars for the audio files

#73
Adding the progress bars for the audio files.
Which will show how much audio you have listened to.
If you pause in between and refresh page you can again start from the same point.
Adding the screenshots  for more details

You have done amazing work but it is not upto date @manoj9896


Use mailables for Laravel 5.3/5.4 Compatibilty

Previously getting error 'only mailables can be queued' when registering new user or sending password reset email.
Two new mailables (CentaurWelcomeEmail and CentaurPasswordReset) added and UserController, RegistrationController and PasswordController updated to use mailables.

This is great!  Thank you!

@rydurham I noticed you added $email to the arguments for CentaurPasswordReset in PasswordController but $email is not used in the view (as it is for the welcome email) so it does not need to be passed to the mailable.
I'm not sure, but I expect the user will get an error because you have not changed the constructor in the mailable class (or declared the $email variable), so it will still only expect a single argument. Correct me if I'm wrong.

Good catch!   I have now removed that reference on the master branch.   Thanks!


init() doesn't work on Block_Controller and subclasses

Goal: to be able to load in custom Requirements (JS and CSS) on a per-Block basis.  With Widgets, this was accomplished by adding an init() function to the WidgetController.  This method does not carry through to Blocks.
This may be related:  when attempting to access {$URLSegment}/block/{$Block-ID}, a 500 error is generated with message:
Call to a member function getController() on a non-object in /var/www/blocks/code/extensions/BlockContentControllerExtension.php on line 32

Such an error does not occur with Widgets, which have a similar ContentController extension.

Hi Sekjal, you can place your requirements calls in your template. I think that's the best approach as it keeps frontend stuff in the frontend. Alternatively you can place them in the forTemplate() method on your Block class.
The controller error might be an issue specific to 3.2. I think we need to give the controller a url segment. You on ss 3.2?

Still running 3.1.13.  Once I get through this next major dev push, I'll look at upgrading.
I'll add the Requirements to the templates temporarily, but I've just made a move system-wide to put them all in the controllers.  If I can figure out how to get init() to work more or less normally, would that be a worthwhile pull request?

What do we gain from being able to use a controller init() method over a
forTemplate() on the model? Is it just consistency with the way it's often
done on pages/contentcontrollers? Might be worth while but not a biggie.
Happy to merge a PR if you send one through.
On Thu, Dec 3, 2015 at 9:53 AM, Ian Walls<EMAIL_ADDRESS>wrote:

Still running 3.1.13. Once I get through this next major dev push, I'll
look at upgrading.
I'll add the Requirements to the templates temporarily, but I've just made
a move system-wide to put them all in the controllers. If I can figure out
how to get init() to work more or less normally, would that be a worthwhile
pull request?
—
Reply to this email directly or view it on GitHub
https://github.com/sheadawson/silverstripe-blocks/issues/38#issuecomment-161430862
.

--
Shea Dawson
Web Developer
LiveSource
mobile +64 20 4028 5647
skype squatchnz
email<EMAIL_ADDRESS>*web *www.livesource.co.nz

Point taken; using forTemplate() does exactly what I'd want init() to do, so why prioritize this over other enhancements?  Perhaps a note in the documentation on how to do this would be a better pull request... I'll work on that and submit.

Sure thing :)
On Fri, Dec 4, 2015 at 3:06 AM, Ian Walls<EMAIL_ADDRESS>wrote:

Point taken; using forTemplate() does exactly what I'd want init() to do,
so why prioritize this over other enhancements? Perhaps a note in the
documentation on how to do this would be a better pull request... I'll work
on that and submit.
—
Reply to this email directly or view it on GitHub
https://github.com/sheadawson/silverstripe-blocks/issues/38#issuecomment-161649515
.

--
Shea Dawson
Web Developer
LiveSource
mobile +64 20 4028 5647
skype squatchnz
email<EMAIL_ADDRESS>*web *www.livesource.co.nz


Preventing goroutine leaks

When job execution times out, the errorc channel is never emptied.
Because of this, the goroutine never finishes.

You are right! thanks!


added next-colors to readme

@iamlite thank you for sharing your project! I like the code quality and JSDoc documentation!


Check tooling available for writing DMPs

This is a subtask of

#1

DMPTool
DMPonline
ARGOS
DMP OPiDOR
Research Data Management Organiser

public instance

See also generic information via Forschungsdaten.info.

While glancing over these notes again, I noticed that DSWizard was missing in the above list. Documentation has moved to https://guide.ds-wizard.org/.
Anyway, I just came here to say that I had a good chat with two from the GFBio DMP team - well aligned in spirit and maybe eventually in action.

Another one to keep in mind: https://www.dmptuuli.fi/

For the purposes of the InvasionBiologyHypotheses project, the overview of the available tooling was good enough to create a first DMP, as per

#1 .

I will probably get back to this later in the context of

#8


[OCU-97] make putil save intermediate results to more than "file://"

Thanks for the patience and the questions.
Replies below and in the RFC

Why do we need this?

security tools are not always idempotent, hence a workflow that failed for whatever reason cannot always be restarted with the same results
deduplication enrichers save things to a database, if you fail at the consumer stage everything then is a duplicate but has not been reported
the ability to restart a workflow from the last successful stage is a core component of any workflow engine

this is the first step at implementing retryable steps

Is it something urgent?
No, future looking for now, necessary after v1.0

How can we prevent the storage backend from being spammed with huge amounts of data?

I have yet to see a security tool that provides huge numbers of data without the tool itself dying trying to export it. We can design for a scale of low Mb.

How is it enabled thoughout a pipeline?
Every single component takes --in and --out parameters, part of the component definition.
These parameters are currently paths.
In the future they should be urls

What are the new methods that will be added?  What are their signatures?
updated the RFC.

writeToDB(dbConnStr, object)error
readFromDBG(dbConnStr)(object,error)

writeToFS(path,object)error
readFromFS(path)(object,error)

What will be the names of the tables where the output will be written?

implementation detail, in the rfc

Do we need to refactor the components or just the SDK?

just the SDK


GitLeaks Task Failed getting Local issuer Certificate on Windows Agent

We have an agent set up to connect to out through our corporate proxy and we are seeing this issue when trying to run Gitleaks.
''' ##[error]unable to get local issuer certificate```
Any help is appreciated.

Hi @cliftonz1 yes, never tested it with a proxy, so I can imagine it has bugs. Can you run the task with a variable called system.debug and set the value to true?
Then please copy/paste the log of the Gitleaks task fully. Thank you!

Can you also provide me with what kind of proxy you are running? It seems that there is a certificate required. I see that the Gitleaks task works with basic proxy stuff. It takes the proxy settings from the agent.

@cliftonz1 ?

@JoostVoskuil  Sorry for the late reply.
We use NetScaler for this issue.
I have also opened a ticket with Microsoft and will update this thread accordingly.

Hiu @cliftonz1 sorry but I don't understand you. Do you have this issue also for other tasks or only Gitleaks?

I updated the last comment,
I am only seeing this issue on Gitleaks, however I am unsure if this is a Gitleaks issue or an agent issue.

So what does Microsoft has to do with it?
Can you provide me with the requested information?


build: fix github runner incorrectly tags failed builds as passed

Fixes WIFI-979
Signed-off-by: Arif Alam<EMAIL_ADDRESS>

Were we using the build.log file? When I download the full raw logs from the build process, I don't see a file with this name but I do see huge files of name 1_build-ecw5211.txt (for instance). These appear to possibly be target specific renames of build.log.
Please check that the raw build logs still contain the full output of the build process.

Confirmed that raw build logs still contain full output of the build process.  The raw build logs are generated by github runner.

merged into pending, Thanks !


Add pwned-passwords-django

https://github.com/ubernostrum/pwned-passwords-django/

Or at least consider adding the CommonPasswordValidator.
https://docs.djangoproject.com/en/3.2/topics/auth/passwords/#enabling-password-validation


Service creation failure backoff-and-retry interval doesnt work for volume errors.

Steps to reproduce:

Initialize swarm on a node.
docker swarm init

Create a node local volume.
docker volume create testvolume

Now, create a service that uses testvolume, but updates it with its own set of volume options.
docker service create --name testservice --replicas=1 --mount type=volume,volume-opt=o=addr=<IP_ADDRESS>,volume-opt=device=:/srv/nfs,volume-opt=type=nfs,source=testvolume,target=/nfs alpine sh -c 'while date; do sleep 5; done'

At the moment, this is allowed. However, this behavior is incorrect since volumes are immutable. So adding a patch such as the following, should error out the task creation for the service.
diff --git a/volume/store/store.go b/volume/store/store.go
index 4f4cffa..3d6a314 100644
--- a/volume/store/store.go
+++ b/volume/store/store.go
@@ -319,6 +319,7 @@ func (s *VolumeStore) Create(name, driverName string, opts, labels map[string]st
 // TODO(cpuguy83): With v2 plugins this shouldn't be a problem. Could also potentially
 // use a connect timeout for this kind of check to ensure we aren't blocking for a
 // long time.
+
 func (s *VolumeStore) checkConflict(name, driverName string) (volume.Volume, error) {
        // check the local cache
        v, _ := s.getNamed(name)
@@ -380,6 +381,8 @@ func volumeExists(store *drivers.Store, v volume.Volume) (bool, error) {
 // If a volume with the name is already known, it will ask the stored driver for the volume.
 // If the passed in driver name does not match the driver name which is stored
 //  for the given volume name, an error is returned after checking if the reference is stale.
+// If there are volume options and there's an existing volume by the given name, then
+// return error. Volumes are immutable, so dont update their options.
 // If the reference is stale, it will be purged and this create can continue.
 // It is expected that callers of this function hold any necessary locks.
 func (s *VolumeStore) create(name, driverName string, opts, labels map[string]string) (volume.Volume, error) {
@@ -398,6 +401,10 @@ func (s *VolumeStore) create(name, driverName string, opts, labels map[string]st
                return nil, err
        }

+       if v != nil && len(opts) > 0 {
+               return nil, errors.New("Volume already exists. Cannot update volume with new options.")
+       }
+
        if v != nil {
                // there is an existing volume, if we already have this stored locally, return it.
                // TODO: there could be some inconsistent details such as labels here

With the patch in (3),
a. service creation will error out, as expected.
b. task creation will continue to be retried, as expected.
However, what is not expected is that there's no significant backoff. Task creation is retried several times per second. Trying the same operation at the same frequency, even if there are repeat failures doesn't seem like the right approach.

9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=0wvseumw41hqhspqlyobqpgka
time="2018-04-24T20:17:37.169003711Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=i4lwcq057314d6xl9jclaxanq
time="2018-04-24T20:17:37.422401740Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=y3cwz256itmn7no24wrd53nqn
time="2018-04-24T20:17:37.980495566Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=kspuaordfrtwxmhursywi1uwr
time="2018-04-24T20:17:38.241111984Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=z4g4ixgzh08jas7ncljbhdvgl
time="2018-04-24T20:17:43.257636689Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=ul5n20f8tqbaavfyvnt9uy406
time="2018-04-24T20:17:43.715472855Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=o4o43wolkycsfivkf0rcg2nsz
time="2018-04-24T20:17:43.968831289Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=ki7i0rfxsbcmb3nszwvegxfz1
time="2018-04-24T20:17:44.429070990Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=82qll18eas5xlk32oba16qf1k
time="2018-04-24T20:17:44.680785829Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=q9vhfh38bk7dsfhs2kn0rsrg1
time="2018-04-24T20:17:45.140651013Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=6i193tuaogjqr27gq8gurokyx
time="2018-04-24T20:17:45.397362558Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=mhscqs933qp8x31olcp0tvz1t
time="2018-04-24T20:17:45.857018316Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=m5062def7kn38l07hhjqxw5dj
time="2018-04-24T20:17:46.114124741Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=0axci90iy5l2q58qdgt4u39h8
time="2018-04-24T20:17:46.569823338Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=z9sutjj4ooypxc45oc5g6roo7
time="2018-04-24T20:17:46.823200870Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=5jvjzu69hyuozlarrant1ttyf
time="2018-04-24T20:17:47.278200943Z" level=error msg="fatal task error" error="create testvolume: Volume already exists. Cannot update volume with new options." module=node/agent/taskmanager node.id=hiou35y4l9lx0jx67sw9mqppq service.id=wn2957oykk9v9th05ledqj25a task.id=1azlxta7v7vvwv235mh1bfz5d

There's 1 second backoff (not exponential) for task retries. https://github.com/docker/swarmkit/blob/master/agent/task.go#L172. However, it's not taking effect.


Update meta balena 5x owa

@klutchell I just pushed the Rebase button and after it the "Versioned source" failing problem appeared. What is the problem?
Could help me understand what happened please?
Thanks!

@owzagj Our versioning checks only work with proper rebases from master, not by merging changes from master into your branch. I suggest performing a rebase from origin/master on the command line and force pushing again.

@klutchell I just pushed the Rebase button and after it the "Versioned source" failing problem appeared. What is the problem? Could help me understand what happened please?
Thanks!

When your PR is out of date you have the option in the GitHub UI to update the branch out of which you created the PR. When you do that make sure you click on the arrow next to the Update branch button and select the Update with rebase option and then click on Rebase branch

Nice @floion, this morning a new commit to the master branch brought me the opportunity to test your suggestion and it worked!
Now what is it waiting for? Jenkins build?
Thanks a lot


Add support for PyMC >= 5 with PyTensor backend

Hi @dfm!
I wanted to play with PPLs during .Astronomy 12, including the latest PyMC version. I figured adding it to exoplanet-core would be a simple way to get familiar with the codebase while doing something useful. A few notes:

I copied the pymc4 directory to pymc and did the changes there, as PyMC no longer uses the version number in the package name.
Most of the changes simply required replacing aesara with pytensor.
It's worth noting that the pymc4 and pymc backends are actually in conflict because PyMC v4 also uses pymc as the package name.

Does it mean it's better to keep pymc5 as the submodule name here for the latest version?
Is there a way to specify that the two "extras" are mutually exclusive in the setup.py? I did not find any documentation on this.

Some tests in pymc_jax_test.py are failing, but they were already failing in pymc4_jax_test.py

Let me know if there is anything I should add/change.
Thank you!

In terms of naming, I'd be happy to just remove the PyMC4 implementation since it was so short lived and then we'll just have pymc3 and pymc which seems about as unambiguous as we're going to get :D

@vandalt — I'm making a few changes to your branch to get the test suite running. Can you pull from your branch before making anymore changes?
Remaining on your end is to remove the compiler flags, and to look into the details of the shape question above. Let me know if there's anything I can do yo help get this over the finish line!

I'm making a few changes to your branch to get the test suite running. Can you pull from your branch before making anymore changes?

Thanks, noted!

Remaining on your end is to remove the compiler flags, and to look into the details of the shape question above. Let me know if there's anything I can do yo help get this over the finish line!

I started reading about the shape question this weekend and ended up doing a deep dive into how the graphs works in PyMC (which I had not really done before). I'll probably have more time to look into this in ~ 2 weeks.

Sounds good - thank you!!

I finally had time to look more into this.

Regarding the Jax part, it was failing because of #90: import was skipped because importing ShapedArray failed, so the op was not defined. Now everything works!

For shape and compiler flags, I left comments in the threads above.

Thanks @vandalt!!


fix presets expansion

Ensure we have the right environment when expanding all fields of preset.
Fixes #4019

changelog?


Theming and per entity theming not working for badges

Checklist

[x] I'm running the latest version of CustomUI (Update guide) or using a specific release that is not marked as "Broken".
[x] I tried to force-refresh (Ctrl+Shift+R / Ctrl+F5) the browser
[x] (Optional, but recommended) I'm using Chrome or tried to reproduce the feature on Chrome.

Browser + Version:
Chrome, 62

CustomUI version:
2017-10-19
Home Assistant release (hass --version):
0.56.2
Problem-relevant configuration.yaml entries:
Everything else works OK (regarding themes and per entity themes). I can set a theme for all of them, however not per entity.
The variables below are ignored by Custom-UI. I didn't try but I am sure all the variables are ignored for the badges (badges can't be themed). Am I doing something wrong?
--label-badge-background-color: white;
--label-badge-text-color: rgb(76, 76, 76);

Sample configuration:
binary_sensor.door_window_sensor_158d0000f125f7:
      state_card_mode: badges
      templates:
        theme: >
            if (entity.state == 'on') return 'statuson'; else return 'default';

statuson:
      label-badge-background-color: '#00cc66'
      label-badge-text-color: rgb(76, 76, 76);

Problem-relevant Home Assistant log entries:
No errors in log.

Any errors from browser Javascript console:
See picture below.

Badges in the top area are indeed beyond CustomUI control. Note that state_card_mode also have no effect on them.
However if you put the sensor in a group and then turn it into a badge using state_card_mode: badges then per-entity theming should work.

Thanks for confirming this. I knew about the possibility to theme badges in groups. I was wondering about the others.
Since you wrote the code and you are extremely familiar with the frontend, would it be possible/an easy job to extend Custom-UI's functionality to cover the badges? I am interested in the per entity theming.

I'm afraid the way the code is built it is not possible.

Thanks again for your answer. That is a shame. I will take a look, maybe I can code something fast.
An unrelated question. Is it intentional to have the Custom-UI badges larger than the standard ones (including the text)?

was wondering about the different sizes as well...

The large size is the "native" size of the badge implementation. In the top row they are sized at 85%.
Do you think the badges are too large?

I would say that it would be nice to have the same size for all badges (unless the size means something).
Maybe it's just my OCD.

We share the same OCD i guess ;) I’d like  smaller, same sized ones better, too.

+1 for smaller badges, or at least the same size as the regular one


STAB mode in V1.1.0 / firmware error?

I tested my setup against px4 firmware v1.1.0 with my fixed wing "big turtle". I have flown this airplane since 14 month without any problems on stablized mode in various firmware developement steps from px4 (I did all the laser landing testing with ZHAW with this aircraft).
On normal flight behaviours (smooth movements on roll and pitch), everything worked fine on my recent test of v1.1.0. So there was everything well.
But I wanted to "rock" the plane respectivly the flight controller and dit some fast movements on roll. Even this was going ok. But in my last few flight seconds, I combined fast rolls with very narrow turns, and then it happended. The plane was going in a fast vertical spin until impact. It was to fast for me to switch to "manual" and got the following result...

Ok. Bad luck, fatal damage.
But I would be very pleased, if you could analyze if there was a px4-firmware funcitonal issue behind. Of course, it could be any technical issue other than the firmware.
log muncher is here:
http://dash.oznet.ch/view/okTSrSHWyK4sjSCbqvP8Cn
Thanks for analyzing! I cannot see what could be the cause here.
Best wishes
Severin

Comment: I dint overstressed the airframe. It was a very narrow turn, but not aerodynamically critical.
I have been flying the airframe since 2.5 years and it was everything in borders of "not critical" in point of view from the aircraft itself. And I have done the same maneuvers with the flight controllers APM and FY-DOS before, and it was going well.

When exactly was it out of control for you? Towards the end you were in STAB for about 1 min (~from 460 to 520). At the time you switched to STAB it went into a fast descent already. Before it hit the ground you gave full rudder (~500). Was this the attempt to stop the spiral?
Could you also post the contents of the preflight and postflight txt files for this log?

I would say out of control was on 432.0 and impact on 434.40.
Maybe you can provide me a flightplot preset to monitor my RC inputs on roll, pitch, throttle and yaw?

One of us is looking at the wrong log then. That's what I have at the end from the log you posted in this issue:

Using the log muncher flight data I can see a clear pilot-induced stall.

I am very sorry Lorenz, I uploaded the wrong file. ....
Your analysis references to a nice and good working high alpha landing from my previous flight.
To not generate to much confusion, I do open a new issue with the correct file!
Best, Severin

@severinleuenberger @LorenzMeier @AndreasAntener The lession that I learnt from this is that a huge spike in the accelerometer does not necessarily mean crash. Obviously also nice belly landing cause these accelerations. However, if you look at the gps speed it's apparent that this was a nice landing.

@tumbili  where do you see a huge spike in the accelerometer in the above chart? Just for me to learn to interpret logs.
And of course, the big turtle is somehow a littlebit a strange aircraft.
Just my 2 cent about "stabilized mode" would be, that it mostly refers to the gyro and accelerometers, and the scaling factor from "airspeed" should only be used, if there is oscillation from to big servo movements. Otherwise its more safe to use stabilized most simple to just level the plane.
Even baro should not be used in stabilized from my point of view. Just leveling the plane and allowing the max bank and pitch angles als defined in the param-list.  But i will reopen this disscussion in the new issue. I do not have access to the sd card now, this evening i can do this.

@severinleuenberger Check the "raw acceleration" chart on logmuncher.


I would like to play mp4 video using memory stream

I have encrypted video which I would like to play chunk by chunk using memory stream
can this be possible with VideoPlayer component?
request if you can make that way, at least play from memory stream

This should be possible using the VideoPlayer.GetStreamForWriting method.
Example when using VideoReader:
// open player stream - specify format, resolution, framerate and pixel format
var playerStream = VideoPlayer.GetStreamForWriting("rawvideo",
    $"-video_size {width}:{height} -framerate {framerate} -pixel_format rgb24",
    out Process playerProcess);

// copy from stream to player stream
video.DataStream.CopyTo(playerStream);

You can use any stream for this, just make sure the FFplay parameters are correct! Make sure to specify the correct video format, resolution, framerate and pixel format. Framerate should be given in format "60/1" for 60fps etc...

Thank you, very much for your answer


Add Badges for unread

http://getbootstrap.com/components/#badges

Moved to https://github.com/fastchat/web/issues/20


feat: Add consortia to collection endpoints for the BE Data Portal API

#TICKET_NUMBER
3641

Reviewers
Functional:
Readability:

Changes

Add consortia[] to the GET /v1/collections response
backend.portal.api.app.v1.collections.actions.get

Add consortia[] to the GET /v1/collections/{collection_id} response
backend.portal.api.app.v1.collections.collection_id.actions.get

Add consortia[] to the GET /v1/collections/index response
backend.portal.api.app.v1.collections.index.get

Add optional consortia[] to the PUT /v1/collections/{collection_id} request and store in the database.

Add consortia[] to the PUT /v1/collections/{collection_id} successful (200) response.

Add consortia[] to the POST /v1/collections/{collection_id} successful (200 201) response.
backend.portal.api.app.v1.collections.actions.post

QA steps (optional)
Notes for Reviewer

Abandoned this as we merged these changes into #3875


UseBootstrapFileProvider throws NullReferenceException

https://github.com/Unleash/unleash-client-dotnet/blob/60ac24c737f0907e072904c96a1e0af46bbda833/src/Unleash/UnleashSettings.cs#L224
This method uses FileSystem property, but it is never initialized, so it is impossible to use the bootstrap using a json file.
For now, I'm using reflection to solve my problem.

Interesting, should probably add a null check with a fallback here the same way UnleashService does when it checks the filesystem property

Exactly! That solves the issue, for now I'm doing that:
    internal class UnleashSettingsInitializeFeaturesConfigurator : IConfigureOptions<UnleashSettings>
    {
        private readonly IConfiguration _configuration;
        private readonly UnleashOptions _unleashOptions;

        public UnleashSettingsInitializeFeaturesConfigurator(IConfiguration configuration, IOptions<UnleashOptions> unleashOptions)
        {
            _configuration = configuration;
            _unleashOptions = unleashOptions.Value;
        }

        public void Configure(UnleashSettings options)
        {
            var enableFeaturesInitialization = _configuration.GetValue("INITIALIZE_FEATURETOGGLE", false);

            if (enableFeaturesInitialization)
            {
                var fileSystemType = Type.GetType("Unleash.Internal.FileSystem, Unleash.Client");
                var fileSystemObj = Activator.CreateInstance(fileSystemType!, Encoding.UTF8)!;

                var systemFileProp = options.GetType().GetProperty("FileSystem", BindingFlags.Instance | BindingFlags.NonPublic)!;
                systemFileProp.SetValue(options, fileSystemObj);

                options.UseBootstrapFileProvider("features.json");
            }
        }
    }


Compact the leveldb database on startup.

It's not at all clear to me that this is the correct thing to do. It does seem to drastically speed up certain operations, but also note (from leveldb's docs): "This operation should typically only be invoked by users who understand the underlying implementation." That set of users does not include me.

OK. We can reopen this later or do something similar at a later date if it's warranted.

The change in this branch is trivial, so you could reasonably delete this branch too, though it's not required.


Add support for IPs masked via a Proxy

If the user is behind a load balancer or proxy, $_SERVER['REMOTE_ADDR'] would be the IP address of the Proxy, not the user. Use the Apache headers first and fall back to $_SERVER['REMOTE_ADDR'] if necessary

@josegonzalez updated to support nginx

For the record, this is what SG uses in it's various codebases (python, php, etc.):
    public function getCurrentIpAddress()
    {
        $ordered_choices = array(
            'HTTP_X_FORWARDED_FOR',
            'HTTP_X_REAL_IP',
            'HTTP_CLIENT_IP',
            'REMOTE_ADDR'
        );
        $invalid_ips = array('<IP_ADDRESS>', '::1');

        // check each server var in order
        // accepted ip must be non null and not in the invalid_ips list
        foreach ($ordered_choices as $var) {
            if (isset($_SERVER[$var])) {
                $ip = $_SERVER[$var];
                if ($ip && !in_array($ip, $invalid_ips)) {
                    return $ip;
                }
            }
        }

        return null;
    }

Don't know what @zackkitzmiller's thoughts on this are though.

@josegonzalez i like that function better, shall I update the PR with that or just reject it and update the source?

You can split off the ip by comma if you'd like before checking if it's invalid and returning.

This looks good, and I'll think it over. We won't be able to merge this in, until there is feature parity with all of the other SeatGeek maintained Client Libraries.

@zackkitzmiller thanks. from taking a quick scan of the libraries, it looks like only the PHP implementation tries to determine the user's IP address? The rest of the libraries provide the IP address an an attribute of the model or part of the constructor.

Yeah, you're right. I'll give this a triple check in the morning.

To note, this should resolve #7

@zackkitzmiller sorry to trouble you, any thoughts or updates?

This looks good, @barchard. Thanks for the PR.

I'll make a release this week.


Existing Azure AD Tenant

Can AAD Connect Cloud Sync be used if an org already has accounts in the cloud? Instructions for doing this with the on-prem version are at https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-install-existing-tenant,
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 70e7def0-aa27-80ba-e2e5-94d83410e1a6
Version Independent ID: 4dd1fd12-2b06-ac78-e40b-a2d440ea181b
Content: What is Azure AD Connect cloud sync.
Content Source: articles/active-directory/cloud-sync/what-is-cloud-sync.md
Service: active-directory
Sub-service: hybrid
GitHub Login: @billmath
Microsoft Alias: billmath

@DeanGross
Thanks for the feedback! We will update you shortly.

@DeanGross
Thank you for pointing this out! I've reached out to our engineering team and will update as soon as possible.

@DeanGross
Thank you for your time and patience throughout this issue, I received a response from our engineering team and will post it below.
Update:
For AAD connect cloud sync, the instructions are documented at Pilot Azure AD Connect cloud sync for an existing synced AD forest, and is also covered under supported typologies.

Based off our supported topologies, this does look like it works if an org already has accounts in the cloud. I'll go ahead and assign this issue to the author to update the docs or provide any additional comments as needed.

Nope, looks good.  Thanks @JamesTran-MSFT
#please-close


Add new D3D12 features to the "Feature support for feature levels" table on MSDN

Is your feature request related to a problem? Please describe.
I'm trying to build a table of feature support caps vs feature level for reporting purposes. I'm leveraging this table on MSDN to understand what D3D features are supported/optional/etc given a feature level. The problem is that this table is incomplete, it's missing new D3D12 features like DXR, VRS, etc.
Current table: https://docs.microsoft.com/en-us/windows/win32/direct3d11/overviews-direct3d-11-devices-downlevel-intro
There is a table here with new features I'd like: https://devblogs.microsoft.com/directx/new-in-directx-feature-level-12_2/
Describe the solution you'd like
I'd like this table to be updated with the new features:
https://docs.microsoft.com/en-us/windows/win32/direct3d11/overviews-direct3d-11-devices-downlevel-intro
Describe alternatives you've considered
N/A
Additional context
N/A

cc @stevewhims

Thanks, @sebmerry! I added the info from the blog post to the topic. The blog posts represents feature level 12_2, whereas the table in the topic represents feature levels 12_2 through 9_3. So I added extra tables rather than trying to extend the existing table.
I did notice that adding this info to the existing table makes it unreadable due to scrollbars appearing (a longstanding issue with tables in our docs). So if we do ever want to add info about these features for the other feature levels, too, then I'll need to refactor some of the columns down into the following section, and that'll be quite time-consuming.
Thanks!
-Steve


update install script

For our reference, see discussion:
https://github.com/conda/conda/issues/4955

Double checking why Travis differs from local installation:
https://github.com/travis-ci/travis-ci/issues/4191

As a workaround, until the conda issue is solved, I have enforced the required version of htslib:
conda install -f htslib=1.3


Fix requirements.txt

Need emcee dev version (or to rewrite the fit examples)

Rewritten fit_examples to not use emcee 3.0.0dev


Wav files

Just a question, can react-player play .wav files?
Thanks!

ReactPlayer just uses <video> for files, so if the browser can play wavs, then ReactPlayer can.


Novatek 98562 Openipc Support

As I have downloaded the Openipc files for NT98562 and need to know how can i flash that files to my Hardware.

Novatek support is broken currently

As i have the hardware and the SDK any way to get support or anything from my side to support for OpenIPC support on  Novatek

You can fix current platform issues if you have developer skills


feat: Add poetry as dependency manager

Added poetry as dependency management. This will allow a more exact dependency versioning and project description in the repository itself. Also, it allows us to split off the dependencies into groups, ex. general, dev, and test., and control which one to install when.
Check Poetry Docs for more information about how to use it.
Also, the requirements.txt file was updated with the version constraints using the command:
poetry export --with test --without-hashes --output .\requirements.txt
Assumptions
For the PR, it was assumed that the version of Python used is 3.10, and for the dependencies, the last compatible version was used.

@acuriel it would be good to know how to use poetry to keep our dependencies updated. I'm new to poetry myself, and maybe some other contributors are also new to it. Maybe add the basic commands to run after a particular set of changes are made?
I'm reading the documentation, but I think having this kind of cheat sheet for us newbies can be useful. Do you agree?

Yep, of course! Should i add it to the README?

Yep, of course! Should i add it to the README?

I guess it would be good on the README. I don't know if there is any convention for documentation on Python repos. If there is, I think we should follow it. Maybe having a docs folder? I leave it to your consideration.
One thing to consider though:

If we should run some poetry commands after making some changes, then this should also be told on the CONTRIBUTING file. Do you agree?

I guess it would be good on the README. I don't know if there is any convention for documentation on Python repos. If there is, I think we should follow it. Maybe having a docs folder? I leave it to your consideration.
One thing to consider though:

If we should run some poetry commands after making some changes, then this should also be told on the CONTRIBUTING file. Do you agree?

As a suggestion, you can make a makefile to run these commands for ease of use.

Hi, @acuriel are you still working on this? No rush, I just want to know a little bit of the status since I plan to add some new implementations soon and this poetry thing sounded really good.

Hi @albexl , yep, i have it ready and set up, but i haven't  updated the readme 😅. Let me do it tomorrow and also to add some reference links and examples so everyone can make a smooth transition

commands

Hmm, I'm not sure if a makefile here would be worth it. We would need to document  the commands, and at the end, the poetry cli is really simple and well documented. Maybe a reference and examples could bw enough

commands

Hmm, I'm not sure if a makefile here would be worth it. We would need to document the commands, and at the end, the poetry cli is really simple and well documented. Maybe a reference and examples could bw enough

👍🏻👍🏻


run on a headless server

Hi, thank you for having released this cool project!
My goal is to run the simulation and training on a headless server (os without GUI). Could I face issues with the gazebo simulator or should I flag something to make it work properly?
Follow-up question: what do you think about running everything inside a docker container? Do you have any tips on which image to use?
Thank you. Regards!

Hi, sure you should be able to run it without a GUI as long as you have access to terminal. Gazebo simulator is already turned off by drfault and you are not required to run Rviz either. The tensorboard should provide you with the course of the training and terminal outputs are also available. However, the KPIs that i have set up here might not always be super representative of how well the model performs. I usually try to also visually determine the progress of the training. But if you add some additional KPIs, visual confirmation is not strictly necessary.
Sorry, have never worked with ROS in docker setup so I cannot help you with that aspect.

I see, thank you for your answer. I'm trying to run the system on a docker container with ubuntu 20.04, ros noetic and cuda. I update the thread as soon as I get everything up.
From a first run, I receive this output
process[gazebo-1]: started with pid [48299]
process[urdf_spawner-2]: started with pid [48302]
process[robot_state_publisher-3]: started with pid [48305]
process[joint_state_publisher-4]: started with pid [48306]
ERROR: cannot launch node of type [rviz/rviz]: rviz
ROS path [0]=/opt/ros/noetic/share/ros
ROS path [1]=/home/mrosano/DRL-robot-navigation/catkin_ws/src/velodyne_simulator/velodyne_simulator
ROS path [2]=/home/mrosano/DRL-robot-navigation/catkin_ws/src/velodyne_simulator/velodyne_gazebo_plugins
ROS path [3]=/home/mrosano/DRL-robot-navigation/catkin_ws/src/velodyne_simulator/velodyne_description
ROS path [4]=/home/mrosano/DRL-robot-navigation/catkin_ws/src/multi_robot_scenario
ROS path [5]=/opt/ros/noetic/share
[INFO] [1672928601.180439, 0.000000]: Loading model XML from ros parameter robot_description
[INFO] [1672928601.186518, 0.000000]: Waiting for service /gazebo/spawn_urdf_model
[ INFO] [1672928601.328079499]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1672928601.329107483]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
[ INFO] [1672928601.554408391]: waitForService: Service [/gazebo/set_physics_properties] is now available.
[ INFO] [1672928601.568448960, 0.207000000]: Physics dynamic reconfigure ready.
[INFO] [1672928601.791623, 0.425000]: Calling service /gazebo/spawn_urdf_model
[ INFO] [1672928602.173754939, 0.544000000]: Laser Plugin: The 'robotNamespace' param was empty
[ INFO] [1672928602.173841720, 0.544000000]: Starting Laser Plugin (ns = r1)
[ INFO] [1672928602.175132927, 0.544000000]: Laser Plugin (ns = r1)  <tf_prefix_>, set to ""
[INFO] [1672928603.477350, 0.544000]: Spawn status: SpawnModel: Successfully spawned entity
[ INFO] [1672928603.478113623, 0.544000000]: Velodyne laser plugin missing <min_intensity>, defaults to no clipping
[ INFO] [1672928603.480406014, 0.544000000]: Velodyne laser plugin ready, 16 lasers
[ INFO] [1672928603.583988398, 0.544000000]: Starting plugin DiffDrive(ns = r1/)
[ INFO] [1672928603.584088024, 0.544000000]: DiffDrive(ns = r1/): <rosDebugLevel> = Debug
[ INFO] [1672928603.585072794, 0.544000000]: DiffDrive(ns = r1/): <tf_prefix> =
...
...
[ WARN] [1672928603.585679791, 0.544000000]: GazeboRosDiffDrive Plugin (ns = ) missing <publishTf>, defaults to 1
[ INFO] [1672928603.586876196, 0.544000000]: DiffDrive(ns = r1/): Advertise joint_states
[ INFO] [1672928603.587973843, 0.544000000]: DiffDrive(ns = r1/): Try to subscribe to cmd_vel
[ INFO] [1672928603.592225340, 0.544000000]: DiffDrive(ns = r1/): Subscribe to cmd_vel
[ INFO] [1672928603.593221874, 0.544000000]: DiffDrive(ns = r1/): Advertise odom on odom
[ INFO] [1672928603.598418598, 0.544000000]: GazeboRosJointStatePublisher is going to publish joint: chassis_swivel_joint
[ INFO] [1672928603.598452058, 0.544000000]: GazeboRosJointStatePublisher is going to publish joint: swivel_wheel_joint
[ INFO] [1672928603.598476248, 0.544000000]: GazeboRosJointStatePublisher is going to publish joint: left_hub_joint
[ INFO] [1672928603.598498880, 0.544000000]: GazeboRosJointStatePublisher is going to publish joint: right_hub_joint
[ INFO] [1672928603.598519501, 0.544000000]: Starting GazeboRosJointStatePublisher Plugin (ns = r1/)!, parent name: r1
[DEBUG] [1672928603.636296702, 0.565000000]: Trying to publish message of type [nav_msgs/Odometry/cd5e73d190d741a2f92e81eda573aca7] on a publisher with type [nav_msgs/Odometry/cd5e73d190d741a2f92e81eda573aca7]
[DEBUG] [1672928603.636364164, 0.565000000]: Trying to publish message of type [sensor_msgs/JointState/3066dcd76a6cfaef579bd0f34173e9fd] on a publisher with type [sensor_msgs/JointState/3066dcd76a6cfaef579bd0f34173e9fd]

[urdf_spawner-2] process has finished cleanly
log file: /root/.ros/log/809b0176-8d04-11ed-a8db-0242ac110003/urdf_spawner-2*.log
Validating
..............................................
Average Reward over 10 Evaluation Episodes, Epoch 1: -72.518953, 0.800000
..............................................

Is this the output I should expect? Does the [urdf_spawner-2] process has finished cleanly message show a normal behavior?
I monitor the training from tensorboard. For now the "Av.Q" goes lower and lower, the "Max. Q" increases, the "Loss" is stable to around 200 since last 150 steps.
Best, Marco

Yes all of that looks like it is working well and all looks like things that I would expect. The urdf spawner thing is related to spawning the robot in Gazebo simulator and that is the expected behavior.
If it has gone through a validation and no errors popped up in the terminal, that should be a good indicator that everything is up and running as it should.

Update: I successfully launched the training process inside a docker container with ros noetic, cuda and all the libraries required but unfortunately it diverges instead of converging to the optimal solution. I launched the script more than 5 times and I trained the model for more than 150 epochs. The behavior is always the same: on tensorboard the average Q value steadily decreases (-700 after 14k steps), the max Q value is really noisy, the loss reached very high values (10k after 14k steps). It seems something is not working properly. Do you have any idea on what the issue could be?

It is good to hear that it worked in headless state.
I guess this is where the issue would arise, as you cannot visually evaluate if everything is working alright.
Some questions that might point out some issues:
Did you try changing the seed?
Do you get any kind of errors or warnings in the terminal?
What does the terminal say?
Are the sensors working properly?
And can you visually see the tensorboard?

The rviz issue shouldn't pose a problem but you can turn off the call to the node by deleting or commenting put the following line: https://github.com/reiniscimurs/DRL-robot-navigation/blob/943186fb7f1890700ce215951e92d5cb92031d14/TD3/assets/multi_robot_scenario.launch#L14
I will try to look ibto this when i have a computer at hand.

Hi, all.
I would like to ask if there has been any updates on the divergence issue Marco was facing.
I am facing a similar issue. The robot does not converge to the optimal solution and the average Q decreases while the loss is vibrating and I also ran the training for around 500-600 steps for multiple hours. I am using a CPU (not CUDA) and it is on a VM - don't know if that's an issue though. Everything seems to be fine though. RViz launches when the training starts. When I run 'gzclient', I get the Gazebo simulation, no errors, only warning I get are 'tf_repeated_data ignoring data with redundant timestamp for frame ...' but I don't think it's related to the training behavior.

Hi,
The usual solution is to try to use different seeds and see what works best for you. Another way is to manipulate the bootstrapping distances in the simulation. See if that helps.
tf_repeated_data usually means that something is being published twice. This is not the expected behavior. I suggest checking the rqt tree to see if there are any anomalies. Also, please write down your setup, the installed branch of the repo and any other information that you could provide.

My setup:
ROS Noetic
Ubuntu 20.04
CPU: The VM uses around 12GB RAM, 4 cores (I'm not using Nvidia Cuda)
I installed the main branch (ROS Noetic + Pioneer robot, the one you talked about in the Medium articles)
Pytorch 1.9.1+cpu
Tensorboard 2.11.2
I don't know if that's the information you're asking about and thank you for poiting about the tf_repeated_data issue, it could be a reason for the bad behavior. Any ideas how to solve it?
I tried changing the seed number and change different things in the code (the exploration, the max_action to slow the robot down when it's heading towards an obstacle, increasing the batch size to the default 100, allow the robot to go backward and letting the sensor sees 360 degrees, increasing the laser_state_dim to 40, etc) but nothing is actually working, although the behavior improved in some trials but most of the time, the robot just heads straight to the obtsacle.
What do you mean by manipulating 'the bootstrapping distances in the simulation'?

That's the rqt_graph

I installed from scratch and trained the model over the weekend on a local system and it seems to work just fine in my setup. So i will rule out a bug in the repository itself.
@adeldennaoui
The ROS topic graph looks fine here. What I would be more interested to see is the rqt tf tree. You can obtain it in rqt with plugins->visualization->tf tree.
The log output to terminal is also something that could show some issues, so please post that as well.
@rosanom
Your terminal log output looks just fine. The only thing I notice is that the urdf_spawner (gazebo_ros/spawn_model) is without an indent (unlike other nodes). I assume this is just a formatting error when pasting it into the comment? Are you also not using cuda as training device?
So far the common things seem to be the use of a "remote" system and perhaps using cpu for training. I will try running the training without cuda and see what that shows.

Here is the tf tree, I don't know if anything's off about it (I took this screenshot before the occurance of the tf_repeated_data error, I'll update it after it happens in case the tf tree changes):

I can confirm that besides the use of a remote system (VM for me), I am training with CPU so it'd be interesting to know if the training works with you using a CPU. In any case, I'll install the Nvidia drivers and the CUDA toolkit on the VM and see if that solves the problem and keep you updated. Thank you!

I tested the training with CPU over multiple runs and it seems to be more unstable, at least with the set seed by default.

However, it failed only in one of those runs, and 2 of them showed obstacle avoidance behavior from early on. I only ran the training for about 3 hours in each case, just to see the trend, and not till convergence.
I also realized @adeldennaoui mentioned that training for only 600 steps and a couple of hours. This is not enough to evaluate the convergence of the model. Often the model will start showing the desired behavior in range between 20 to 40 epochs and converge at around 100 epochs. So if the robot is not rotating on the spot (which insinuates that no learning is done whatsoever), it might just not have had enough time to learn. I would suggest training the model for at least 20 epochs, then see if some obstacle avoidance and navigation behavior can be seen and then make a decision. If robot just spins around and does nothing, simply restart the training. If at least some behavior is seen, just continue the training.
@adeldennaoui The TF issue might be caused by some synchronization. Are you using any method to speed up the training? If not, is your simulator capable of running in real time in your VM?

An update: so I started the training process at around 2pm and it's circa 9:30am where I live (that's more than 19 hours since the training started) and the robot has done 34 epochs and the picture shows that the training seems to be successful, the average Q is starting to increase, the reward reward during the validation is starting to increase and become positive (it used to be only negative up the around the 25th epoch) and the behavior of the robot improved drastically but that's despire the tf warnings - very happy about that but it is unfortunate that it took my VM that much time (I used the default seed too).

@adeldennaoui It is good that we can see that it actually works after the amount of epochs I would expect. However, 19 hours for 34 epochs is really a long time. I suspect that the VM you are using simply does not have enough resources to run the Gazebo simulation in real-time and there is a significant slowdown. If you look at your wall time it is around 19 hours, however the actual ROS execution time is below 6 hours in real time. This is also suggested by the TF errors, as the publishing frequencies do not align. You might want to check how well ROS runs on your VM and try to improve the situation there.


NodeBB Integration Client

Integrate Connect with NodeBB. @christiansmith and I will pair program on this.

The code for this will live here:
https://github.com/anvilresearch/nodebb-plugin-sso-anvil-connect


Minor rendering bug for multiline private message at end of feed

It feels like there's some missing padding in the yellow text at the bottom here:

@timabbott As paddingStart is 8dp, i added paddingBottom to also 8dp. It will be  better to see. :)

@zulipbot claim-issue


Automatic Roling

Automatic roling from a spreadsheet not implemented.

Try to make it a splash command with user customisability and discord error messages.


how to get the "TID" from my java application when using  skywalking-agent

Please answer these questions before submitting your issue.

Why do you submit this issue?
[x] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?

Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.

Requirement or improvement

Please describe about your requirements or improvement suggestions.

Please read docs of OSS before asking, https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/Application-toolkit-trace.md

ths
---Original---
From<EMAIL_ADDRESS>Date: Fri, Jul 3, 2020 11:26 AM
To<EMAIL_ADDRESS>Cc: "Author"<author<EMAIL_ADDRESS>Subject: Re: [apache/skywalking] how to get the "TID" from my java application when using  skywalking-agent (#5018)
Please read docs of OSS before asking, https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/Application-toolkit-trace.md
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


Changed the window title to something more appropriate

When using the SDL version to run locally, changed the window title to show more dedication to the 32blit :-)

Oh wow, TinyDebug harks back a while! Good catch, thank you!


feat(rules): Add settings & query parameters

Coverage decreased (-0.1%) to 75.922% when pulling c07dcdc0e2e013bbc41ed52d0f3491032abba118 on feat/rules_parameters into fa4b3f72396f748f5f9a975503d31aac41203d90 on master.


KAFKA-2657; Kafka clients fail to start if one of broker isn't resolved by DNS

Thanks for the patch. The implementation looks good to me. But I am not sure if the motivation is strong. The exception is only thrown if the client can not resolve broker's address during initialization. Note that DNS resolution failure is relatively rare and usually happen if the address is wrong. Why no just let the client fail so that operator can immediately look into the issue?
You can also ping a committer for comments.

We ran into this issue today. We were doing a rolling upgrade of existing brokers, when one of our services that's a producer was rolled out as well. This prevented our producer service from starting up when the rest of the brokers were fine. It would be nice to have this patch put into both 0.8.2.x and 0.9.x.

Thanks for the PR. Sorry for not getting to this sooner. The issue that the PR tries to solve was resolved via https://github.com/apache/kafka/commit/c4bbf342432291220242ad4177fd72a959ddcb94 so feel free to close this PR (we can't do it ourselves).


Zeta Testnet node started but not syncing

Which environment are you setting a node for?
Testnet
Does your machine match the technical requirements?
Yes
Have you completed the setup?
Yes
Which method are you using?
Snapshot
Did you build your binary from source or download it from the Releases page?
None
Please, share the contents of config.toml
proxy_app = "tcp://<IP_ADDRESS>:26658"

# IMPORTANT! Replace the placeholder below with your actual moniker
moniker = "tenderly-zeta-testnet-node"

fast_sync = true
db_backend = "pebbledb"
db_dir = "data"
log_level = "info"
log_format = "plain"
genesis_file = "config/genesis.json"
priv_validator_key_file = "config/priv_validator_key.json"
priv_validator_state_file = "data/priv_validator_state.json"
priv_validator_laddr = ""
node_key_file = "config/node_key.json"
abci = "socket"
filter_peers = false

[rpc]
laddr = "tcp://<IP_ADDRESS>:26657"
cors_allowed_origins = []
cors_allowed_methods = ["HEAD", "GET", "POST", ]
cors_allowed_headers = ["Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time", ]
grpc_laddr = ""
grpc_max_open_connections = 900
unsafe = false
max_open_connections = 900
max_subscription_clients = 100
max_subscriptions_per_client = 5
experimental_subscription_buffer_size = 200
experimental_websocket_write_buffer_size = 200
experimental_close_on_slow_client = false
timeout_broadcast_tx_commit = "10s"
max_body_bytes = 1000000
max_header_bytes = 1048576
tls_cert_file = ""
tls_key_file = ""
pprof_laddr = "localhost:6060"

[p2p]
laddr = "tcp://<IP_ADDRESS>:26656"

# IMPORTANT! Replace the placeholder below with your actual external IP address
external_address = "<IP_ADDRESS>:26656"

seeds = "5aaa51a3b9465a32f7f6c9df1d46d4bfcc16aecb@<IP_ADDRESS>:26656"
persistent_peers = "ea3ca43db14f35f4fc1dfc883e78e7162103047f@<IP_ADDRESS>:26656,9a2e8d08a0f61b318c608350e080c000c97e9cf2@<IP_ADDRESS>:26656"
upnp = false
addr_book_file = "config/addrbook.json"
addr_book_strict = true
max_num_inbound_peers = 40
max_num_outbound_peers = 10
unconditional_peer_ids = ""
persistent_peers_max_dial_period = "0s"
flush_throttle_timeout = "100ms"
max_packet_msg_payload_size = 1024
send_rate = 5120000
recv_rate = 5120000
pex = true
seed_mode = false
private_peer_ids = ""
allow_duplicate_ip = false
handshake_timeout = "20s"
dial_timeout = "3s"

[mempool]
version = "v0"
recheck = true
broadcast = true
wal_dir = ""

size = 5000
max_txs_bytes =<PHONE_NUMBER>
cache_size = 10000
keep-invalid-txs-in-cache = false
max_tx_bytes = 1048576
max_batch_bytes = 0
ttl-duration = "0s"
ttl-num-blocks = 0

[statesync]
enable = false
rpc_servers = "<IP_ADDRESS>:26657,<IP_ADDRESS>:26657"
trust_height = 0
trust_hash = ""
trust_period = "168h0m0s"
discovery_time = "15s"
temp_dir = ""
chunk_request_timeout = "10s"
chunk_fetchers = "4"

[fastsync]
version = "v0"

[consensus]
wal_file = "data/cs.wal/wal"
timeout_propose = "3s"
timeout_propose_delta = "500ms"
timeout_prevote = "1s"
timeout_prevote_delta = "500ms"
timeout_precommit = "1s"
timeout_precommit_delta = "500ms"
timeout_commit = "5s"
double_sign_check_height = 0
skip_timeout_commit = false
create_empty_blocks = true
create_empty_blocks_interval = "0s"
peer_gossip_sleep_duration = "100ms"
peer_query_maj23_sleep_duration = "2s"

[storage]
discard_abci_responses = false

[tx_index]
indexer = "kv"
psql-conn = ""

[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
max_open_connections = 3
namespace = "tendermint"

Please, share the contents of app.toml
minimum-gas-prices = "1.0azeta"
pruning = "nothing"
pruning-keep-recent = "100"
pruning-keep-every = "0"
pruning-interval = "10"
halt-height = 0
halt-time = 0
min-retain-blocks = 0
inter-block-cache = true
index-events = []
iavl-cache-size = 781250
iavl-disable-fastnode = true

[telemetry]
service-name = "tss"
enabled = true
enable-hostname = true
enable-hostname-label = true
enable-service-label = false
prometheus-retention-time = 10
global-labels = [ ]

[api]
enable = true
swagger = true
address = "tcp://<IP_ADDRESS>:1317"
max-open-connections = 1000
rpc-read-timeout = 10
rpc-write-timeout = 0
rpc-max-body-bytes = 1000000
enabled-unsafe-cors = false

[rosetta]
enable = false
address = ":8080"
blockchain = "app"
network = "network"
retries = 3
offline = false

[grpc]
enable = true
address = "<IP_ADDRESS>:9090"

[grpc-web]
enable = true
address = "<IP_ADDRESS>:9091"
enable-unsafe-cors = false

[state-sync]
snapshot-interval = 0
snapshot-keep-recent = 2

[evm]
tracer = ""
max-tx-gas-wanted = 0

[json-rpc]
enable = true
address = "<IP_ADDRESS>:8545"
ws-address = "<IP_ADDRESS>:8546"
api = "eth,net,web3,txpool,personal,debug"
gas-cap = 25000000
evm-timeout = "5s"
txfee-cap = 1
filter-cap = 200
feehistory-cap = 100
logs-cap = 10000
block-range-cap = 10000
http-timeout = "30s"
http-idle-timeout = "2m0s"
allow-unprotected-txs = false
max-open-connections = 0
enable-indexer = false

[tls]

Additional details
Node is started, we waited for long time for rpc server to be started, to check latest block number at first.
Then node opened rpc server, but latest block number is staying at 5,594,795 and not moving.
Version of node is v17.0.0.
Here are [
debug logs
](url)
Often invalid tx result code can be observed, also Blockpool has no peers

@nebojsa-ristovic can you share the value of :26657/status (Comet BFT RPC).

Is this node an archive node built from one of our filesystem snapshots? If so, know that it will take 8.5 hours for the node to start the first time after unpacking the archive snapshot on a high performance NVMe server.
There are no log messages indicating progress, and one might think that the node is frozen or stalled. It is important to let the node finish initializing after applying the snapshot.

Thanks everyone, node started processing with no action taken, it took some time like @CryptoFewka said.


胶囊网络碰到一些问题

你好，想请教一下一个问题：
当我用胶囊网络的时候，我发现损失在不断的下降，但是，精度却在不断的降低？这让我很疑惑。
==========  Start training  ==========
Train Epoch: 1 [640/4000 (16%)]	Loss: 0.808406
Train Epoch: 1 [1280/4000 (32%)]	Loss: 0.807249
Train Epoch: 1 [1920/4000 (48%)]	Loss: 0.805759
Train Epoch: 1 [2560/4000 (63%)]	Loss: 0.803572
Train Epoch: 1 [3200/4000 (79%)]	Loss: 0.800597
Train Epoch: 1 [3840/4000 (95%)]	Loss: 0.796495
Train Epoch: 1 [4000/4000 (100%)]	Loss: 0.795066
macro metrics: [p: 0.4085, r:0.2570, f1:0.2254]
micro metrics: [p: 0.2570, r:0.2570, f1:0.2570]
Train Epoch: 2 [640/4000 (16%)]	Loss: 0.789447
Train Epoch: 2 [1280/4000 (32%)]	Loss: 0.782183
Train Epoch: 2 [1920/4000 (48%)]	Loss: 0.773557
Train Epoch: 2 [2560/4000 (63%)]	Loss: 0.762742
Train Epoch: 2 [3200/4000 (79%)]	Loss: 0.750204
Train Epoch: 2 [3840/4000 (95%)]	Loss: 0.735720
Train Epoch: 2 [4000/4000 (100%)]	Loss: 0.731115
macro metrics: [p: 0.1102, r:0.1210, f1:0.0532]
micro metrics: [p: 0.1210, r:0.1210, f1:0.1210]
Train Epoch: 3 [640/4000 (16%)]	Loss: 0.714159
Train Epoch: 3 [1280/4000 (32%)]	Loss: 0.695173
Train Epoch: 3 [1920/4000 (48%)]	Loss: 0.674588
Train Epoch: 3 [2560/4000 (63%)]	Loss: 0.652236
Train Epoch: 3 [3200/4000 (79%)]	Loss: 0.628999
Train Epoch: 3 [3840/4000 (95%)]	Loss: 0.607024
Train Epoch: 3 [4000/4000 (100%)]	Loss: 0.601029
macro metrics: [p: 0.0100, r:0.1000, f1:0.0182]
micro metrics: [p: 0.1000, r:0.1000, f1:0.1000]
Train Epoch: 4 [640/4000 (16%)]	Loss: 0.581706
Train Epoch: 4 [1280/4000 (32%)]	Loss: 0.566093
Train Epoch: 4 [1920/4000 (48%)]	Loss: 0.553648
Train Epoch: 4 [2560/4000 (63%)]	Loss: 0.543705
Train Epoch: 4 [3200/4000 (79%)]	Loss: 0.536618
这是什么情况呢？

你好，这是因为过拟合了。
你跑的是demo里采样数据吧，这个数据量太小了，很容易过拟合。


Irrelevant errors after successfully uploading a DOCX file using drag & drop

Describe the Bug
As of the implementation of the WYSIWYG Docx Import which was launched in v23.06.2, working with DOCX files has been considerably easier; but there are still some issues.
In spite of being said that "These hacks are unsupported customizations meant as unofficial workarounds", I think fixing this issue can be a huge improvement.
After uploading the DOCX file, the contents of the file is migrated successfully but some irrelevant errors pops up. (screenshots included)
Steps to Reproduce

Open a new page.
Drag & drop a DOCX file.
The migration is successful but there are some errors shown on the screen that are frustrating.

Expected Behaviour
I expect the migration to be clean and easy. After uploading the DOCX file, I don't want to encounter any errors.
Screenshots or Additional Context

Browser Details
Google Chrome Version 104.0.5112.102 (Official Build) (64-bit) On Windows 10
Exact BookStack Version
v23.06.2
PHP Version
No response
Hosting Environment
Ubuntu 20.04 VPS, deployed on docker.

As per the warning you quoted from our hacks site, these things are not considered officially supported parts of BookStack and therefore I'm not looking to service issues for these in this repo.
The warnings are advised within the considerations for the hack. Additionally, there will always be issues and edge-cases in this kind of functionality. There are reasons this is provided as a hack instead of being built into the platform.


Disable message object to send after execution

Hey is there a way to disable the message object to be sent after using the <prefix>jsk py  await _ctx.send("hello world")
If not via code then any changes in the source code
Help appreciated
Thanks in advance

Either do jsk py "hello world" or jsk py py await _ctx.send("hello world") and None

Cool thanks for your help


Possible issue with katz_centrality

I ran into this one with LightGraphs (since I used your katz code there):
julia> g = evolving_graph(Int, String)
Directed EvolvingGraph (0 nodes, 0 edges, 0 timestamps)

julia> add_edge!(g,1,2,"t1"); add_edge!(g,2,3,"t1"); add_edge!(g,1,3,"t1"); add_edge!(g,3,4,"t1")
Directed EvolvingGraph (4 nodes, 4 edges, 1 timestamps)

julia> EvolvingGraphs.katz_centrality(g,0.4)
4-element Array{Tuple{Int64,Float64},1}:
 (4,0.3136745577315065)
 (3,0.4391443808241091)
 (2,0.4893323100611502)
 (1,0.6850652340856103)

but the same graph has a different Katz centrality in NetworkX:
In [11]: z = nx.katz_centrality(g,0.4)

In [12]: z
Out[12]:
{1: 0.3164769948422847,
 2: 0.44306779277919855,
 3: 0.620294909890878,
 4: 0.5645949587986359}

It appears that I can get LightGraphs (and presumably you could get Evolving Graphs) to yield the same results as NetworkX by transposing the adjacency matrix: A = A'. (That is, in LightGraphs, instead of using the outbound adjacency matrix, use the inbound.)
NetworkX is generally how I prove the correctness of my algorithms. It's entirely possible that they've got it wrong here, but my first instinct is to blame my own code :)
Would you mind helping me check this out?
Ref: NetworkX
Ref: https://github.com/JuliaGraphs/LightGraphs.jl/issues/151
cc: @jpfairbanks

Hi @sbromberger ,
Thanks for the question. In Katz's original paper (http://people.cs.vt.edu/badityap/classes/cs6604-Fall13/readings/katz-1953.pdf), indeed, the transpose of the adjacency matrix A is used.  I think this is why NetworkX use the transpose of A. However, in many recent research papers, for example see (http://centaur.reading.ac.uk/19357/1/Coomunicability_accepted.pdf), both A and its transpose are considered.

the solution to the equation (I - αA)x = ones(n) is called broadcasting centrality which measures how effectively nodes can broadcast information; and

the solution to the equation (I - αA')x = ones(n) is called receiving centrality.

Actually, another implementation of katz_centrality in EvolvingGraphs gives both broadcasting and receiving centrality. See https://github.com/weijianzhang/EvolvingGraphs.jl/blob/master/src/algorithms/katz_centrality.jl#L44
Best wishes,
Weijian

Ah, cool - that makes sense and it's trivial to provide both types of centrality. Thank you very much for the explanation!

I've just added mode= to LightGraphs. Thanks again.

You are welcome :)

@weijianzhang, there seems to be an inconsistency here L19 and L59
One uses v = S\v and the other uses v = S*ones. Is the formula on line 59 computing the inverse of S or something?

@jpfairbanks

Is the formula on line 59 computing the inverse of S or something?

Yes, you are right.
S =  (spI + e^(-β*Δt)*S)*(spI + α*A) - spI is an approximation of
S =  (spI + e^(-β*Δt)*S)*inv(spI - α*A) - spI,
where inv(spI - α*A) = spI +  α*A +  α^2*A^2 +  α^3*A^3 + ...

Is that close enough using only two terms of the analytic expansion?
I don't usually think of 1+x as close to 1/1-x
On Sep 8, 2015 4:51 AM, "Weijian Zhang"<EMAIL_ADDRESS>wrote:

@jpfairbanks https://github.com/jpfairbanks
Is the formula on line 59 computing the inverse of S or something?
Yes, you are right.
S = (spI + e^(-βΔt)S)(spI + αA) - spI is an approximation of
S = (spI + e^(-βΔt)S)inv(spI - αA) - spI,
where inv(spI - αA) = spI + αA + α^2A^2 + α^3A^3 + ...
—
Reply to this email directly or view it on GitHub
https://github.com/weijianzhang/EvolvingGraphs.jl/issues/5#issuecomment-138483257
.

@jpfairbanks well, I wouldn't say the approximation is very good. But in many cases, we want to just "restrict our attention to walks using at most one edge per time point".  This approximation was suggested by the original paper. "A Matrix Iteration for Dynamic Network Summaries", Peter Grindrod and Desmond J. Higham,  http://epubs.siam.org/doi/abs/10.1137/110855715


[zh] add page: /zh/docs/tasks/administer-cluster/safely-drain-node/ ,fix 24097

add file: content/zh/docs/tasks/administer-cluster/safely-drain-node.md
translate the file from its english version

close  #24097

change is done.
thanks for your comment.

/approve


EXP: switch to using calc-full-gather.py

This PR switches slainte over to using calc-full-gather.py from https://github.com/ctb/2024-calc-full-gather / https://github.com/sourmash-bio/sourmash_plugin_branchwater/issues/187, which does not run a whole new gather with a picklist, but instead calculates the columns starting from the fastgather output.
This has the advantage of being lower memory and faster, per https://github.com/sourmash-bio/sourmash/issues/2950. This is especially true for large nasty rumen samples, ugh.
Before this gets merged, we would need to fix calc-full-gather to work with multiple databases, among perhaps other things.

This PR also triggered https://github.com/sourmash-bio/sourmash/pull/2952 :)


JsonMap.keys() paths are not escaped to enforce literal interpretation

Many methods within JsonMap indirectly rely the paths (segments) returned by keys(). If such a path segment matches the conditions that have a special interpretation they will be misunderstood by the get(String) because they have to be taken literally.
For example, when a path is foo.{x}.bar the segments from splitting at . are foo, {x}, bar. {x} is then misunderstood as x because of the special syntax to "quote" member access by enclosing a path in curly braces.
Similar issues might surface in other API methods.
To circumvent this either any method returning a path must "escape" it to indicate literal interpretation or the special syntax needs to be adjusted to avoid such clashes in other ways.

Fixed in 1.1. release :)


Support ROCm 5.2

Tasks

[ ] Read ROCm Release Notes.
[ ] Update AMD driver in Jenkins test infrastructure (ask @kmaehashi).
FlexCI

[ ] Update .pfnci/schema.yaml and .pfnci/matirx.yaml
[ ] Backport.

Wheel Distribution

[ ] Update cupy-release-tools.
[ ] Backport.
[ ] Update Jenkins job configuration for ROCm wheel build (other_cupy_release_builder) (ask @kmaehashi)

Wheel Package Detection

[ ] Add the package to the duplicate detection.
[ ] Backport.

Documentation

[ ] Add new wheel package to the Installation Guide and README.md. Update requirements in the installation guide.
[ ] Backport.
[ ] Add new wheel package to the website.

[ ] Implement or create an issue to support new features, if applicable.

Closing as ROCm 5.3 is released. #7092

xref #6828


Add lint command flag to ignore circular references for polymorphic definitions.

This works well with patched libopenapi

I am assuming once you bump the version of libopenapi, the failing test will pass.

@daveshanley That's mostly it. Looking forward to the release.

I think you need to run go mod tidy

@daveshanley Yes, for some reason go.sum was not up to date.

You're welcome @daveshanley. It would be great you'd mention these flags in the documentation.

Done.


Passwords not complex enough

I try to run deploy.ps1, enter the two passwords and because NOTHING is said about the password rules, I simply take my favorite password.
But then:
New-AzureRmResourceGroupDeployment : 10:00:26 - Resource Microsoft.Sql/servers 'mydriving-dbserver-7up3tmzzuc7n6'
failed with message 'Password validation failed. The password does not meet policy requirements because it is not
complex enough.'
In C:\dev\MyDriving\scripts\PowerShell\deploy.ps1:144 Zeichen:16
+ ... ployment2 = New-AzureRmResourceGroupDeployment -Name "$DeploymentName ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

New-AzureRmResourceGroupDeployment : 10:00:27 - Resource Microsoft.Sql/servers 'mydrivingdbserver-7up3tmzzuc7n6'
failed with message 'Password validation failed. The password does not meet policy requirements because it is not
complex enough.'
In C:\dev\MyDriving\scripts\PowerShell\deploy.ps1:144 Zeichen:16
+ ... ployment2 = New-AzureRmResourceGroupDeployment -Name "$DeploymentName ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

This is crappy.  What rules does my password have to obey?

strong passwords: https://msdn.microsoft.com/library/ms161962.aspx
powershell scripts are for administrators and there is expectation, that admins knows something about security

This is an assumption - nice, of course. But each platform has its own specifications as to what they consider a strong password. So, a short text BEFORE asking for two passwords are surely helpful.

short text with hint added to deployment script


134 dev lab equipment digital twin

In this PR, below changes are made:

Modified DoE Agent to align with the latest practice of agent development
Created VapourtecExecution Agent to schedule the experiment execution on vapourtec based on digital twin state
Created Vapourtec Agent to control the execution of the scheduled experiment
Created HPLC Agent to monitor the HPLC report and upload that to the knowledge graph
Created HPLCPostPro Agent to post-process the raw HPLC report
Created chemistry_and_robots package to support all knowledge graph interactions required by the above agents
Ontology TBoxs: OntoReaction, OntoDoE, OntoLab, OntoVapourtec, OntoHPLC, OntoBPR

The fleet of agents is tested in silico and in actual deployment for dummy reaction.

I'm not suggesting this should be changed in this PR, but for future reference: Instead of
"... %s ... %s ..." % (a, b)
please use the much more elegant, readable, and contemporary f-strings
f"... {a} ... {b} ..."
(use double curly brackets if you need the actual curly bracket characters).

Thank you very much for the feedback, I will follow this practice in future development. All the rest points are addressed in the latest commits.


Fix windows cache path issue

Fixes issue where (on windows) the character ":" is not allowed in a folder path.

Hi @gregorhutterer, thanks for the PR!
I see the relevant issue from issue #66
We'll investigate & test it along with upcoming features to improve this SDK.

Hi @gregorhutterer / @schabibi1, is there any way how we can help with the release of this small fix? Or is there at least some workaround as it's a big blocker for work with important parts of the content?