No apparent flushing for TrackMessage or TrackException

I'm trying to connect this to Laravel 9.x with logging.
I was not able to get any log messages to come into app insights until I modified Mondago\ApplicationInsights\ApplicationInsights and called flush() in the trackMessage and trackException methods:
https://github.com/mondago/laravel-azure-application-insights/blob/main/src/ApplicationInsights.php#L243
https://github.com/mondago/laravel-azure-application-insights/blob/main/src/ApplicationInsights.php#L259
I'm not sure when / how the flushing is supposed to happen unless its added here? Obviously if someone is wanting to just use:
Log::debug('debugging');

They don't want to have to also inject the client class and call flush do they? I can see flush was used in a few of the other tracking types, but not here. So, I'm not sure if I'm missing something about the expected usage or if this is a bug?

I can't get this package to log either.  @drmmr763 , can you give more detail about how you got it to log?  Are you saying that you created your own branch and added a couple of extra "flush" statements and after doing that, the package worked as expected?

So I modified the vendor package and added the flush method call just to experiment and see what was going on.
It did work, but I'm not using the package because of this issue. I hoped a new release would come out with a fix.

Hey @ossentoo and @drmmr763
Sorry for the delayed reply. Delighted to hear our package is being used. We're sorry that it isn't quite working as you'd expect.
We appreciate the time taken to open an issue as this increases our awareness to real world usage.
You can see from our readme section on middleware that we suggest using our TrackRequest middleware. The reason we don't flush until after the request is sent is because each flush makes a HTTP request to my understanding. This would slow down your response, dependent on how quickly Microsoft can respond to each event. This grows linearly the more logging you have. Instead, we queue up the telemetry (exceptions, logs, traces, request, response) and send them all in one batch from TrackRequest after the response has been sent to the browser (so long as you are running in a FastCGI environment. See Laravel's documentation on this)
We'd happily consider a PR that made flushing on every event configurable however. I'm sure other consumers would love to have the option. Perhaps always_flush or something similar, what do you both think?

Hi @ingalless
Thank you for explaining I will experiment with that. I do a lot of console based processing of information that throws exceptions and I have a lot of logging related to that. Because it is console based and not API / HTTP requested based, I'm assuming that the TrackRequest middleware might never fire for me in that context, is that right?
Additionally, in my use case, i actually have a background processing server that consumes and executes queue jobs that doesn't have any http service on it at all. It just processes queues. Then another HTTP server that processes api requests for the same application. In that environment I'm assuming the queue processor will just always be queueing telemetry on that machine, but never flush due to the fact a TrackRequest will never be handled.
An always flush or something a little smarter such as buffer based would be a great option. Looks like they did something similar here
https://github.com/app-insights-php/monolog-handler/pull/4

@drmmr763 Thank you for the additional context, really useful and interesting stuff going on here!

I do a lot of console based processing of information that throws exceptions and I have a lot of logging related to that. Because it is console based and not API / HTTP requested based, I'm assuming that the TrackRequest middleware might never fire for me in that context, is that right?

You're right that TrackRequest won't be doing anything if it's console based. In our use-cases internally we are calling the trackEvent method for logging which does call flush every time.

In that environment I'm assuming the queue processor will just always be queueing telemetry on that machine, but never flush due to the fact a TrackRequest will never be handled.

Again, your assumption seems correct.
There are events we can listen to for Artisan commands, Jobs and Queues, which seem like a good place to flush. Which do you think fits your use case?

https://laravel.com/docs/10.x/scheduling#events
https://laravel.com/docs/10.x/artisan#events
https://laravel.com/docs/10.x/queues#job-events (These feel user-land. Might not be relevant but included just in case)

Hi @ingalless thanks for taking the time to reply.
So I have kind of homebrewed my own desired functionality for now. I based it on the Monolog repo and also took some inspiration from your TrackRequest.
What I ended up with is this:
TrackRequest terminable middleware extremely similar to what you have. There's some differences to get things like the userID showing in Azure -- for some reason the session stuff wasn't loading in the user. It may be because I'm using stateless oauth passport based authentication.
For TrackRequest You'll see its basically a combination of the TrackRequest and the ApplicationInsights classes into one single class. This does flush on every request - but since it is terminable it is fairly async imho and hasn't had an affect on performance so far.
Then for logging I went for an all in one approach. A monolog-azure handler. This benefits from having Monolog handle buffering and queuing, and flushing automatically for me. So, I haven't had to solve that since they are doing it higher up the chain.
Hopefully this helps give you some ideas on what I ended up needing / wanting.
TrackRequest.php
<?php
// inspired by mondago/laravel-azure-application-insights

namespace App\Http\Middleware;

use AppInsightsPHP\Client\Client;
use AppInsightsPHP\Client\TelemetryData;
use ApplicationInsights\Channel\Contracts\Request_Data;
use ApplicationInsights\Channel\Contracts\User;
use Closure;
use Illuminate\Http\Request;
use Symfony\Component\HttpFoundation\Response;

/**
 * TrackRequest
 */
class TrackRequest
{
    /**
     * @var Client $telemetryClient
     */
    private $telemetryClient;

    /**
     * @var Request_Data $requestData
     */
    private $requestData;

    /**
     * @var $requestProperties
     */
    protected $requestProperties;

    /**
     * @param Client $telemetryClient
     */
    public function __construct(Client $telemetryClient)
    {
        $this->telemetryClient = $telemetryClient;
    }

    /**
     * Handle an incoming request.
     *
     * @param \Illuminate\Http\Request $request
     * @param \Closure                 $next
     * @return mixed
     */
    public function handle(Request $request, Closure $next)
    {
        $this->requestData = $this->telemetryClient->beginRequest(
            'app ' . $request->getMethod() . ' ' . $request->getUri(),
            $request->fullUrl(),
            $this->getRequestStartTime(),
        );

        return $next($request);
    }

    /**
     * Post request processor
     * @param Request  $request
     * @param Response $response
     * @return void
     */
    public function terminate(Request $request, Response $response)
    {
        $this->setDefaultRequestProperties($request);
        $this->setUserId($request);

        $this->telemetryClient->endRequest(
            $this->requestData,
            $this->getRequestDurationTime(),
            $response->getStatusCode(),
            $response->isSuccessful(),
            $this->getRequestProperties(),
            null
        );

        $this->telemetryClient->flush();
    }

    /**
     * Set the user information for an authenticated user
     * @param Request $request
     * @return void
     */
    public function setUserId(Request $request)
    {
        if ($request->user() == null) {
            return;
        }

        /** @var \App\Models\User $user */
        $user = $request->user();

        $userContext = $this->telemetryClient->getContext()->getUserContext() ?? new User();
        $userContext->setAuthUserId($user->id);
        $userContext->setId("$user->first_name $user->last_name");
        $this->telemetryClient->getContext()->setUserContext($userContext);
    }

    /**
     * Start time of the request
     * @SuppressWarnings(PHPMD.Superglobals)
     * @return mixed
     */
    private function getRequestStartTime()
    {
        return $_SERVER['REQUEST_TIME_FLOAT'];
    }

    /**
     * Approximate duration of the request
     *
     * @return float|integer
     */
    private function getRequestDurationTime()
    {
        return (microtime(true) - $this->getRequestStartTime()) * 1000;
    }

    /**
     * Extract default properties from request
     *
     * @param Request $request
     * @return void
     */
    private function setDefaultRequestProperties(Request $request) : void
    {
        $this->addRequestProperty('url', $request->fullUrl());
        $this->addRequestProperty('querystring', $request->getQueryString());
        $this->addRequestProperty('ip', $request->ip());
        $this->addRequestProperty('client-ip', $request->ip());
        $this->addRequestProperty('user-agent', $request->userAgent());
        $this->addRequestProperty('secure', $request->secure());
        $this->addRequestProperty('referer', $request->server->get('referer'));
        $this->addRequestProperty('method', $request->method());

        if ($request->route()) {
            $this->addRequestProperty('route', $request->route()->getName());
        }
    }

    /**
     *  Set request property
     *
     * @param string $key
     * @param mixed  $value
     * @return void
     */
    public function addRequestProperty(string $key, $value) : void
    {
        $this->requestProperties[$key] = $value;
    }

    /**
     * Information from the request
     *
     * @return array
     */
    private function getRequestProperties()
    {
        return $this->requestProperties;
    }
}

AppInsightsLogHandler.php
<?php

namespace App\Logging;

use AppInsightsPHP\Client\Client;
use AppInsightsPHP\Client\TelemetryData;
use ApplicationInsights\Channel\Contracts\Message_Severity_Level;
use ApplicationInsights\Channel\Contracts\User;
use Monolog\Formatter\FormatterInterface;
use Monolog\Handler\AbstractProcessingHandler;
use Monolog\Logger;
use Monolog\LogRecord;

/**
 * AppInsightsLoggingHandler
 */
class AppInsightsLoggingHandler extends AbstractProcessingHandler
{
    /**
     * @var Client
     */
    private $telemetryClient;

    /**
     * @param Client  $telemetryClient
     * @param integer $level
     * @param boolean $bubble
     * @suppressWarnings(PHPMD.BooleanArgumentFlag)
     */
    public function __construct(Client $telemetryClient, int $level = Logger::DEBUG, bool $bubble = true)
    {
        parent::__construct($level, $bubble);
        $this->telemetryClient = $telemetryClient;
    }

    /**
     * @param array $records
     * @return void
     */
    public function handleBatch(array $records) : void
    {
        parent::handleBatch($records);
        $this->reset();
    }

    /**
     * @return void
     */
    public function reset() : void
    {
        $this->telemetryClient->flush();
    }

    /**
     * @return void
     */
    public function close() : void
    {
        $this->reset();
    }

    /**
     * @param LogRecord $record
     * @return void
     */
    protected function write(LogRecord $record) : void
    {
        switch ($record['level']) {
            case Logger::DEBUG:
            case Logger::INFO:
                $level = Message_Severity_Level::INFORMATION;

                break;
            case Logger::NOTICE:
                $level = Message_Severity_Level::VERBOSE;

                break;
            case Logger::WARNING:
                $level = Message_Severity_Level::WARNING;

                break;
            case Logger::ERROR:
            case Logger::ALERT:
            case Logger::EMERGENCY:
                $level = Message_Severity_Level::ERROR;

                break;
            case Logger::CRITICAL:
                $level = Message_Severity_Level::CRITICAL;

                break;
        }

        $formattedRecord = $this->formatter->format($record);

        $message = $formattedRecord['message'];
        $properties = \array_merge(
            [
                'channel' => $record['channel'],
                'datetime' => ($record['datetime'] instanceof \DateTimeInterface) ?
                    $record['datetime']->format('c') : $record['datetime'],
                'monolog_level' => $record['level_name'],
            ],
            $formattedRecord['context']
        );

        if (TelemetryData::message($message, $properties)->exceededMaximumSize()) {
            return;
        }

        if (auth()->hasUser()) {
            /** @var \App\Models\User $user */
            $user = auth()->user();

            $userContext = $this->telemetryClient->getContext()->getUserContext() ?? new User();
            $userContext->setAuthUserId($user->id);
            $userContext->setId("$user->first_name $user->last_name");
            $this->telemetryClient->getContext()->setUserContext($userContext);
        }

        if (isset($record->context['exception'])) {
            $this->telemetryClient->trackException($record->context['exception']);
            return;
        }

        $this->telemetryClient->trackMessage($message, $level, $properties);
    }

    /**
     * @return FormatterInterface
     */
    protected function getDefaultFormatter() : FormatterInterface
    {
        return new ContextFlatterFormatter();
    }
}


Consistently case Snowflake and spec references in permissions

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/1350
Originally created by @tayloramurphy on 2019-12-03 20:17:44

While iterating on https://gitlab.com/meltano/meltano/issues/1153 I felt it was important to ensure consistency with the casing for both the permissions coming from Snowflake and the permissions coming from the spec.

View 5 previous comments from the original issue on GitLab


password value still viewable in page source!

Hi,
Good effort! thank you. But main concern is that the real password value is still traceable in page source when not clicked on the 'eye' icon.
Also, this control cannot be used when this field is added to a 'Quick view form'
Regard
Wim

Hello,
Thank you for your feedback. I'm pleased to inform you that the issue has been fixed in the latest version (v1.1.0).

Hello,
Thank you for your feedback. I'm pleased to inform you that the issue has been fixed in the latest version (v1.1.0).
You can download it from the following link. https://marketplace.bevercrm.com/pcf-controls/PasswordField
We appreciate your help in bringing this to our attention.
Please let us know if you encounter any other issues or have any further questions or concerns.
Best Regards,
Hovhannes
Bever

As for the Quick View Form, this is actually a limitation of Dynamics 365, you can't add a control to this form.


Add support for downloading media files and large files.

Crawlab can't download media files and large files (e.g. jpg files, mp3 files, gif files, zip files and so on.)
A viable approach is to add one type of node. Let's say, MediaWorker. The links of the files (e.g. https://aaa.bbb.com/ccc.mp3) will pass to the MediaWorker node. Then, the media_worker will download the file. MediaWorker node should be able to handle some situation that may occur when downloading large file, such as connection timeout. MediaWorker node may base on some multithreading download library like the aria2.
I believe downloading files (especially image and video files) is a great demand that many appreciate. Though it won't be a easy task to develop such a feature, I'm still looking forward to it.

Are you talking about the configurable spiders?

Are you talking about the configurable spiders?

No, I mean can crawlab download files from the links that spiders yielded?
For example, let's crawl this website: https://www.scientificamerican.com/podcast/60-second-science/
We can get the article title and the corresponding mp3 link,. For example, the title is "Espresso May Be Better when Ground Coarser" and the mp3 link is https://flex.acast.com/www.scientificamerican.com/podcast/podcast.mp3?fileId=D4CC60BE-C5F9-4F8E-B503BCEDE92498B9
What I want is that the mp3 link will pass to the MediaWorker node, and then, the MediaWorker will download this mp3.

Alright, I guess what you are trying to do is a spider workflow that passes crawled data from the a seed spider to another that consumes them. This is something Crawlab doesn't natively support. However, what you can do is to run a spider implemented with a message queue so that two spiders share the same MQ and can pass data via it. If you don't fully understand, you can add my Wechat account "tikazyq1" to discuss further.


Access to calling curl through JF CLI based on the JFrog Platform URL

Is your feature request related to a problem? Please describe.
Currently, the JFrog CLI allows us to perform a number of tasks to maintain the JFrog Platform and the associated applications having the credentials stored away safely.
However not all services which are provided through the REST API which are also fully accessible through the JFrog CLI. This may result in credential information being leaked through build pipeline logs if the REST API's are invoked and insufficient care is given ir order to avoid presenting the information.
Examples of this are that it is possible to create new release bundles, but getting a list of existing release bundles require The REST API's to be called directly.
Describe the solution you'd like to see
I would like to be able to call the REST API from the JFrog CLI based on the JFrog Platform URL instead of being restricted to do so based on the "jf rt curl..." and the "jf xr curl..." which limit which endpoints are available.
An example of this could be:
jf curl -XPOST "/lifecycle/api/v2/promotion...
Describe alternatives you've considered
The alternatives would be to either be limited by which commands have been implemented in the JFrog CLI, or to using the JFrog REST API specifying credentials directly and hoping no credentials are leaked into the build logs.

This would be a great addition!

jf curl would be an invaluable addition to the JFrog CLI


Type annotations for ast module.

Type annotations - beginning with ast.
@tomlarkworthy

LGTM, merge?


feat: include json-ld

fixes #648

It looks like the current implementation only supports a single json-ld property in the metadata block (last one wins?). I think we should add support for multiple, as there are several examples of pages that have more than one. Or can you combine multiple schema objects into a single script tag?

It looks like the current implementation only supports a single json-ld property in the metadata block (last one wins?). I think we should add support for multiple, as there are several examples of pages that have more than one. Or can you combine multiple schema objects into a single script tag?

in any metadata. I assume this will be managed in the metadata sheet anyways.... and we can then still add more
eg with a property json-ld-<name> and create tags like <script data-name="<name>" type="application/ld+json">...

let's keep it simple for now, we can add support for multiple named ones later. i really don't think that's necessary, especially as all the current mechanisms (head.html and client-side) are still there and are working.

@rofe Using an array of JSON-LDs is valid


Update submodule ecl to latest Thu Apr  9 20:22:48 UTC 2020

- ecl in PX4/Firmware (e46683632deb64be80b8e6bdd3305c23231791f2): https://github.com/PX4/ecl/commit/47624a0f021e2c187cc4763ad829afe2aa4fb11a
- ecl current upstream: https://github.com/PX4/ecl/commit/fa5a00d871682e9a4ce7cad292187c24fd01ed8e
- Changes: https://github.com/PX4/ecl/compare/47624a0f021e2c187cc4763ad829afe2aa4fb11a...fa5a00d871682e9a4ce7cad292187c24fd01ed8e

fa5a00d 2020-04-08 kritz - Refactor velocity reset (#788)

@bresch @kamilritz any additional testing needed here?

@dagar maybe just the usual sanity flight tests, but I'm quite confident that it's all good. This was just a non-functional cleanup

This came in separately.


OCPQE-17717 support inheritance rule when parsing ocp build data

https://issues.redhat.com/browse/OCPQE-17717
in latest ocp build data e.g. https://github.com/openshift-eng/ocp-build-data/blob/openshift-4.14/releases.yml, new attribute names are introduced i.e. suffixed with symbol ! e.g. advisories!
xref: https://art-docs.engineering.redhat.com/assemblies/#inheritance-rules
for most important attributes: advisories, release_jira, refenrence_releases, we need to check whether the attribute with suffix ! exists, if yes, get its value for current release, if not, get unsuffixed attribute value, otherwise, get the same attribute value from parent release.

/assign @ming1013  @wewang58

/approve


the project authority management question

Rancher versions:
rancher/server or rancher/rancher:
rancher/agent or rancher/rancher-agent:
Infrastructure Stack versions:
healthcheck:
ipsec:
network-services:
scheduler:
kubernetes (if applicable):
docker 17
centos 7.3
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
single node rancher
Kubernetes 1.8.4
Steps to Reproduce:
1 new user developer
2 add developer k8s-cluster
Name | Role |   |
-- | -- | -- | --
admin | cluster-owner |
developer2 | cluster-member |
developer1 | cluster-member |
developer | cluster-member
3  add developer project owner
Default Admin ( admin )本地用户 | Project Owner | Yesterday at 2:58 PM |
-- | -- | -- | --
  | developer ( developer )本地用户 | Project Owner | 1:41
4 developer enter rancher
enter dev-test project
deploy app OK，first day
but the second day when I enter dev-test project
deploy app not OK. show errors below：
deployments.apps is forbidden: User "u-hxf6r" cannot create deployments.apps in the namespace "dev-test": clusterrole.rbac.authorization.k8s.io "p-87xb9-namespaces-edit" not found
--
Results:
delete the developer in the cluster and project ,then add the user developer in the cluster and project again ,will can deploy app ok

Hi @huangyanhong, I can not reproduce your issue. It should be working in latest version. Can you try v2.0.4, please?

Please feel free to re-open it if you still see this issue.

rancher 2.20 have the same questions


support one-to-many model for the leaf node feature

What would you like to be added:
support one-to-many model for the leaf node feature
Why is this needed:

/assign @qiuming520


Support for AsynApi 2.4.0

Is it possible to update the project to support AsyncApi 2.4.0 spec?
Than you in adavance!!!!

Hi,
Is there any updates related to this issue. Currently there is Async API 3.0.0 available. It would be good to have project up to date.
BR,
Marcin


syscall: support O_SYNC flag for os.OpenFile on windows

The current implementation of os.OpenFile function for windows does
not use the O_SYNC flag. This means that even if the user has set the
O_SYNC flag, the os.OpenFile call will ignore the SYNC flag. This
change adds _FILE_FLAG_WRITE_THROUGH, the equivalent of
O_SYNC flag on Linux.
Fixes #35358.

I couldn't find any of the existing tests and I don't know how to write a test for a syscall. I'm looking for help in writing the tests for this PR.

能不能退订？？？
GopherBot @.***> 于 2021年4月22日周四 07:53写道：

Message from Ian Lance Taylor:
Patch Set 9:
(6 comments)
Please don’t reply on this GitHub thread. Visit golang.org/cl/235198
https://go-review.googlesource.com/c/go/+/235198#message-5570ee668e271dfecefa6b3d906d2fdb33dea010
.
After addressing review feedback, remember to publish your drafts
https://github.com/golang/go/wiki/GerritBot#i-left-a-reply-to-a-comment-in-gerrit-but-no-one-but-me-can-see-it
!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/pull/39245#issuecomment-824437384, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AMOANU7BAMTJNARNX7WMTE3TJ5QPTANCNFSM4NJVXCQA
.


Disable StringAPI test on OS stdlib

These tests verify fixes in the stdlib post-5.9, which means that they fail when tested on older versions of the stdlib.
Fixes rdar://114581543

@swift-ci Please test and merge


Large number in creating shapes

When using rsweep with large number for vertex coord, for example, 500 for x. It will panic:
panicked at 'failed to create instance', truck-rendimpl\src\shaperend.rs:86:14

fn create_test_solid() -> Solid(){

 let vertex = builder::vertex(Point3::new(500.0, 0.0, 100.0 as f64));
        // sweep the vertex along a circle
        let circle: Wire = builder::rsweep(
            // the reference to the vertex
            &vertex,
            // a point on the axis
            // Point3::new(d.offset as f64 + d.radius as f64, 0.0, 0.0),
            Point3::new(500.0 as f64, 0.0, 0.0),
            // the direction of the axis
            Vector3::unit_y(),
            // If the absolute value is no less than 2π radian, a closed shape will be generated.
            Rad(7.0),
        );
        let disk = builder::try_attach_plane(&vec![circle]).unwrap();
        // sweep the circle along a circle
        let mut solid = builder::rsweep(
            // the reference to the wire
            &disk,
            // a point on the axis
            Point3::origin(),
            // the direction of the axis
            Vector3::unit_z(),
            // If the absolute value is no less than 2π radian, a closed shape will be generated.
            Rad(d.angle as f64),
        );
        solid
}

Thank you very much for your report.
We investigated the cause of the problem, and it seems that the Newton method is not converging with the required accuracy when finding the texture coordinates of the boundary curve.
We already know the ad-hoc solution, but I am still trying to decide the best choice for the overall design. We will try to come to a conclusion in a little while.

The following two actions have been taken.

Convergence judgment of Newton's method has been changed to one using derivatives (the judgment condition has been weakened to a feasible accuracy).
The meshing accuracy of simple-shape-viewer is now set by the relative error from the bounding box.

We have saved the results of modeling with the given function in large-torus.json. One can test this issue by launching simple-shape-viewer and dragging and dropping the large-torus.json file. Since it is essentially a tessellation bug, CI only tests the tessellation part.

Now, v0.3 is released. This issue should be closed.


Would you please uniform the version number in git tag and package.json?

Hello, I'm a intern from cdnjs. I want to send pull request to add your library, and it will auto update based on version number in git tag. However, the number should be the same in package.json. So would you please uniform the version number in git tag and package.json? Or you call tell me which number I should follow. Thank you.

Ok. My bad. I fixed the release version. Thanks

OK, Thanks for your collaboration. : )

Excuse me, the cdnjs maintainer want me to ask you that would you mind adding a dist folder for distribution files?

@patopitaluga Sorry for bothering you. I just want to make sure you will notify this new issue I mentioned above. Thank you. :)

@RobertoGonzalez would you mind to take a look? Thanks. cc @sufuf3

@PeterDaveHello Hi! Are you referring to the dist folder or tagging a new version? I see there's a few extra commits that didn't get a proper release.

Hi @RobertoGonzalez, I'm referring the dist folder and it'll be great if we can use the common naming schema, rename the minified file to readremaining.jquery.min.js instead of output.min.js :)

@PeterDaveHello I upped the version to 1.0.1 with those changes. Let me know if it works!

Awesome! Thanks @RobertoGonzalez

Thank you very much.


Upgrade to reactjs-components 0.14.0-beta.38

Reviewer: @Poltergeist

We should make sure not to merge our own version update PRs when updating modules, as this could potentially introduce breaking changes.
We through it offline and made sure that nothing was broken from this update.

Also ran integration tests:


Remove backslash escaping in 2024-08-24 post.

When rendered by Jekyll, some of these are harmless but several of the ones inside link URLs break the links in the rendered output currently live on tritag.ca.
Removing these backslashes should make these links work.

@mdruker at your convenience, ptal?

Not sure what kramdown should have done there, but in any case the Docs output seems suboptimal. I've filed an internal bug against Google Docs.

I've filed an internal bug against Google Docs.

The fix for this bug has now been deployed.


Limit fast domain failover

What changed?
Limit fast domain failover

Why?
A fast domain failover with large load may cause OOM in the service

How did you test it?

Potential risks

Coverage decreased (-0.1%) to 68.847% when pulling 924ecfdda12a865397c2c7a2db04bed3f0a5bdd5 on limit_failover into 63bdb5d7a04285cf57992c574639c9d911c7001a on master.


Images hosted using Cloudflare images and CDN with broken links

Thanks for a quick startup using the docker image. Currently testing and many sites comes out great with this tool.
We are building tools for an human elephant conflict mitigation foundation.
Our backend are using Cloudflare Images/CDN and they may be delivered as JPG, AVIF or WEBP, not sure what the reason is but all image links are broken as shown in the partial snippet from the PDF below generated from the frontpage:

Have tested with the same results:

2.1.0
2.0.0
1.5.2

Switched to https://github.com/gotenberg/gotenberg and it works with all files including css, js and images.

Found the issue. When HTML dumped I had relative paths to assets and therefore broken.
I'm sorry. Should I delete this issue?

Ah, thanks for getting to the bottom of this!
Good luck with your project :)


NOT Working in INTERNET RESTRICTED COUNTRY :'(

This is real good project and I've been using this over 3 years I think. My country has been blocked the internet and can't access the christitus.com domain. Before update I can use with the github project link I think. But don't have that option in this new update. Can you make that happen with the new update or please provide the link for the restricted countries. Thanks.

Hey @ZinK0.
When you go to Known Issues in the ReadMe you will get redirected to the new documentation of WinUtil. There you will find an execution command that should suit your needs. here the direct link to the section I am referencing.
Hope that helps!

@MyDrift-user ,
Thanks. It really help me. I forget to check the documentation and only check from the website. That's why I haven't seen.
Thank you.


How to retrieve the sound data for upload

Great project!
I have one request, though, which is to have the ability to retrieve the wav file data without saving to a file. I would like to upload the data directly to a server.
Another issue is that in the documentation, the saveSound function is in the SoundRecorder object, yet in the examples the standalone function "save" is called. SoundRecorder.saveSound in undefined. Am I missing something here?
Note: I'm using this in a node environment with react.

any update here?

@SirCameron @analizabongat
you could try use the _getBuffers()
https://github.com/processing/p5.js-sound/blob/master/src/soundRecorder.js#L213
maybe this function should be exposed and documented.

Can someone give an example of this working, I require exactly the same thing as they do, actually.
I have a website and I need to work out how to save recordings on to a database, like firebase.

@SirCameron @b2renger @analizabongat
Have any of you found a way that definitely works, yet?

Please check out the getBlob method which has been added recently https://p5js.org/reference/#/p5.SoundFile/getBlob
I hope this isn't way too late to be helpful !


Better Renaming for Trait Members

Hello! This is a suggestion, not a bug report
Currently, the RA doesn't have complete support for trait renaming. Specifically, if I rename a trait method (within the trait itself), we will only rename instances where a trait object is used (I believe), but will instantly run into trouble, as our own implementation of that trait will not have their methods renamed.
I believe that our own implementations should also get renamed -- additionally, I think if I rename a method which is from an implementation of a trait, and I own the trait, we should also rename that method on the trait. That may be harder to implement.
Thank you all for all your hard work!

Sounds like a duplicate of #4201 if I'm not missing something.


Set project with renv?

I'll do this after when its closer to final publication (to help with reproducibility)


Plan to support external data ingestion functions

People will want to ingest data into Dask from a variety of other databases.  For example, here is a small gist someone posted on twitter for a function to ingest from ElasticSearch:
https://gist.github.com/rolando/f4e912aadc8447463209d4fe672c260c
How do we collect these?  I have no desire to hold every possible combination within the dask repository itself, but at the same time there should be some way to curate solutions like this.
What are some options?

Put everything in dask/dask
Manage small repositories like dask/dask-elasticsearch
Create a list of pointers to other projects and put them in documentation
Just let people google for "how to query elasticsearch with dask"

Looks like the author put it on pypi, which is another fine way to handle this.
https://pypi.python.org/pypi/dask-elasticsearch/

https://pypi.python.org/pypi/dask-avro

cc @rolando

cc @martindurant

@mrocklin thanks for keeping me in the loop.
I've been using those little snippets for a while but now required to package it to have it available for another project. Fun fact: I was using a small Elasticsearch instance and was faster to load the search results into Dask and do aggregations than doing it on ES.
The option 1. can definitely be a maintenance burden for core developers. Option 2. brings more visibility to third-party projects.
I'm OK with transferring the projects under Dask organization. I think those small projects could be considered in incubating stage and if they become popular, feature complete and stable, then they can be considered for merging into Dask codebase.

I like the idea of "incubating" related packages and scripts, although a "contrib" area in the main repo, or a contrib repo by itself would have a similar effect. Getting more people to contribute and collaborate must be a good thing.
The important point is for the team not to become responsible for the correctness of contributed code, and to let users know that these might be written to work only for some specific use case, etc.

CCing @kszucs who had similar thoughts about a third-party Marathon deployment package

I like odo's approach: https://github.com/blaze/odo/blob/master/odo/init.py#L26, but the plugins needs to be highly related like cluster backends (SGE, Mesos) or databases/file systems (like s3 of hdfs already) and stable enough. BTW I hate importing dask_marathon/dask_mesos/ :)
Importing from dask or distributed would guarantee the consistent API across plugins, and hopefully decrease the learning curve. It also needs a clear style guide though.
I'd keep these integrations/plugins in separated repos (learning the whole dask/distributed codebase can be frightening), but under dask org (helps discovering).

Are there good ways to maintain a contrib namespace in Python?
On Fri, Feb 3, 2017 at 10:36 AM, Krisztián Szűcs<EMAIL_ADDRESS>wrote:

I like odo's approach: https://github.com/blaze/odo/
blob/master/odo/init.py#L26, but the plugins needs to be highly
related like cluster backends (SGE, Mesos) or databases/file systems (like
s3 of hdfs already) and stable enough. BTW I hate importing
dask_marathon/dask_mesos/ :)
Importing from dask or distributed would guarantee the consistent API
across plugins, and hopefully decrease the learning curve. It also needs a
clear style guide though.
I'd keep these integrations/plugins in separated repos (learning the whole
dask/distributed codebase can be frightening), but under dask org (helps
discovering).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1959#issuecomment-277278935, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszBjlpPEtyLKmSG5bTaQBLHqKMmWLks5rY0ligaJpZM4L13GY
.

There are ways but I wouldn't call them good

I wrote up a quick demo of how we could do this with namespace packages: https://github.com/jcrist/namespace_modules.
Pros:

Slightly prettier import syntax:

from daskit import elastic_search, other_stuff
# vs
import dask_elastic_search
import dask_other_stuff

Namespaces are kind of nice for organizing similar things

Matplotlib does this with the mpl namespace, so there's some precedence here.

Cons:

Slight potential for getting issues in main dask repo about sub-packages that we don't maintain (since the top-level namespace would point to this repo). This probably isn't a big deal though.

May be more effort than it's worth

Either way, I think we should have a docs page with a list of packages, similar to what scikit-learn does.

I think a guideline should be to use the dask namespace for somewhat well-maintained packages. Otherwise there is the risk of polluting dask namespace with half-baked unmaintained modules. It won't be nice to have two or more contrib packages for the same task (i.e. daskit.elasticsearch, daskit.esreader, daskit.fastelasticsearch, daskit.espy4, etc).

If this is already a really big space of subprojects, maybe it is worth looking at how Jupyter manages/solves this problem.
cc @minrk

I think separate repos/packages is a good idea. Being liberal about including such projects on the dask GitHub org is one path to visibility, as is maintaining a good list in your docs. Presence on the org doesn't need to mean that every package is maintained by the same people (examples orgs: zeromq, jupyterhub), but you may take care to indicate this for contributed packages.
Assuming they are separate packages:
I like the idea of namespace packages a lot, but they have been an occasional source of frustration in practice. We considered using namespace packages for Jupyter when we were planning the big split, but ultimately decided against it, going with discrete packages. That doesn't mean it was the right decision, though.
Cons of namespace packages:

People can be confused by import errors of what appear to be subpackages ("I have dask, why don't I have dask.ec2?"). ImportErrors of a top-level package are more searchable.
It seems to be possible to corrupt namespace packages during installation. We still get some bug reports about one or two of our dependencies that use the backports namespace appearing to be unavailable and requiring reinstall. I still don't know how this is happening.

Another possible option (depending on API requirements) is to use  entrypoints. We've started using this in a few places in Jupyter to provide things like custom nbconvert exporters and it's been a nice way for additional packages to make themselves known, but it assumes plugins provide a consistent API.
One option:

use dask.whatever namespace package for 'official' packages
incubating/community packages are standalone packages
community packages can be 'promoted' to dask.thing if that's deemed desirable.

As to a doc listing, PR ( https://github.com/dask/dask/pull/2395 ) seems relevant.

I'm going to close this.  Folks can use normal mechanisms, include in docs, or use aggregation libraries like intake where appropriate.


Low success rate on donwloading laion400m

Hey there, @rom1504,
I have been trying to download laion400m using the scripts from an EC2 instance m5n.8xlarge and the success rate is quite poor.
I am getting a success rate of 10 images for 10k requests with the default command in the README.
Any idea why I am doing wrong ?
Best,
T.C

Did you set up knot resolver?
Please share a wandb link so we can see the error cause
On Sat, Feb 3, 2024, 12:04 PM thomas chaton @.***>
wrote:

Hey there,
I have been trying to download laion400m using the scripts from an EC2
instance m5n.8xlarge and the success rate is quite poor.
I am getting a success rate of 10 images for 10k requests with the default
command in the README.
Any idea why I am doing wrong ?
Best,
T.C
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437V625Q3GBYBWJJOTKDYRYKTHAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTMNBSHA4DCMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Oh interesting. I haven't. Let me try again. What's knot resolver ?

I am getting errors when trying to install knot resolver too.
⚡ ~ wget https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb
--2024-02-03 11:16:00--  https://secure.nic.cz/files/knot-resolver/knot-resolver-release.deb
Resolving secure.nic.cz (secure.nic.cz)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘secure.nic.cz’
⚡ ~ sudo dpkg -i knot-resolver-release.deb
sudo: unable to resolve host ip-10-192-12-27: Temporary failure in name resolution
dpkg: error: cannot access archive 'knot-resolver-release.deb': No such file or directory

Here are the normal logs. Looks like wandb had a Network error (TransientError), entering retry loop
⚡ ~ img2dataset --url_list the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ --input_format "parquet"\
>          --url_col "URL" --caption_col "TEXT" --output_format webdataset\
>            --output_folder laion400m-data --processes_count 32 --thread_count 128 --image_size 256\
>              --save_additional_columns '["NSFW","similarity","LICENSE"]' --enable_wandb True
Starting the downloading of this file
Sharding file number 1 of 32 called /teamspace/studios/this_studio/the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/part-00000-5b54c5d5-bbcf-484d-a2ce-0d6f73df1a36-c000.snappy.parquet
0it [00:00, ?it/s]File sharded in 1294 shards
Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!
wandb: Currently logged in as: thomas-chaton. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.16.2
wandb: Run data is saved locally in /teamspace/studios/this_studio/wandb/run-20240203_111216-t4t3ohoz
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run woven-microwave-1
wandb: ⭐ View project at https://wandb.ai/thomas-chaton/img2dataset
wandb: 🚀 View run at https://wandb.ai/thomas-chaton/img2dataset/runs/t4t3ohoz
wandb: Network error (TransientError), entering retry loop.
1it [04:07, 247.25s/it]worker  - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 10000
17it [04:11,  1.61s/it]wandb: Network error (TransientError), entering retry loop.
22it [04:14,  1.16it/s]wandb: Network error (TransientError), entering retry loop.
24it [04:15,  1.61it/s]worker  - success: 0.008 - failed to download: 0.992 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 83 - count: 20000
worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 124 - count: 30000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 165 - count: 40000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 207 - count: 50000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 245 - count: 60000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 285 - count: 70000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 326 - count: 80000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 366 - count: 90000
worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 406 - count: 100000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 447 - count: 110000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 487 - count: 120000
worker  - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 528 - count: 130000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 569 - count: 140000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 609 - count: 150000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 650 - count: 160000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 690 - count: 170000
worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 731 - count: 180000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 772 - count: 190000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 812 - count: 200000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 853 - count: 210000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 894 - count: 220000
worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 934 - count: 230000
worker  - success: 0.002 - failed to download: 0.999 - failed to resize: 0.000 - images per sec: 42 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 975 - count: 240000
28it [04:21,  1.13s/it]worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 40 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1004 - count: 250000
worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 40 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1038 - count: 260000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 40 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1071 - count: 270000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 40 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1111 - count: 280000
31it [04:26,  1.32s/it]worker  - success: 0.008 - failed to download: 0.992 - failed to resize: 0.000 - images per sec: 39 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1134 - count: 290000
worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 39 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1173 - count: 300000
worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 39 - count: 10000
total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1207 - count: 310000
32it [04:51,  8.13s/it]worker  - success: 0.037 - failed to download: 0.963 - failed to resize: 0.000 - images per sec: 35 - count: 10000
total   - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 1133 - count: 320000

I see you're using 32 processes and 128 threads per process.
That might be too much for the machine you're using, try to decrease
As for knot resolver, please follow their doc for instructions on how to
install it for your distribution
On Sat, Feb 3, 2024, 12:18 PM thomas chaton @.***>
wrote:

Here are the normal logs. Looks like wandb had a Network error
(TransientError), entering retry loop
⚡ ~ img2dataset --url_list the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/ --input_format "parquet">          --url_col "URL" --caption_col "TEXT" --output_format webdataset>            --output_folder laion400m-data --processes_count 32 --thread_count 128 --image_size 256>              --save_additional_columns '["NSFW","similarity","LICENSE"]' --enable_wandb TrueStarting the downloading of this fileSharding file number 1 of 32 called /teamspace/studios/this_studio/the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta/part-00000-5b54c5d5-bbcf-484d-a2ce-0d6f73df1a36-c000.snappy.parquet0it [00:00, ?it/s]File sharded in 1294 shardsDownloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)!wandb: Currently logged in as: thomas-chaton. Use wandb login --relogin to force reloginwandb: Tracking run with wandb version 0.16.2wandb: Run data is saved locally in /teamspace/studios/this_studio/wandb/run-20240203_111216-t4t3ohozwandb: Run wandb offline to turn off syncing.wandb: Syncing run woven-microwave-1wandb: ⭐ View project at https://wandb.ai/thomas-chaton/img2datasetwandb: 🚀 View run at https://wandb.ai/thomas-chaton/img2dataset/runs/t4t3ohozwandb: Network error (TransientError), entering retry loop.1it [04:07, 247.25s/it]worker  - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 1000017it [04:11,  1.61s/it]wandb: Network error (TransientError), entering retry loop.22it [04:14,  1.16it/s]wandb: Network error (TransientError), entering retry loop.24it [04:15,  1.61it/s]worker  - success: 0.008 - failed to download: 0.992 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 83 - count: 20000worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 124 - count: 30000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 165 - count: 40000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 207 - count: 50000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 245 - count: 60000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 285 - count: 70000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 326 - count: 80000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 366 - count: 90000worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 406 - count: 100000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 447 - count: 110000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 487 - count: 120000worker  - success: 0.002 - failed to download: 0.998 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 528 - count: 130000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 569 - count: 140000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 609 - count: 150000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 650 - count: 160000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 690 - count: 170000worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 731 - count: 180000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 772 - count: 190000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 812 - count: 200000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 853 - count: 210000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 894 - count: 220000worker  - success: 0.007 - failed to download: 0.993 - failed to resize: 0.000 - images per sec: 41 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 934 - count: 230000worker  - success: 0.002 - failed to download: 0.999 - failed to resize: 0.000 - images per sec: 42 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 975 - count: 24000028it [04:21,  1.13s/it]worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 40 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1004 - count: 250000worker  - success: 0.004 - failed to download: 0.996 - failed to resize: 0.000 - images per sec: 40 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1038 - count: 260000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 40 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1071 - count: 270000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 40 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1111 - count: 28000031it [04:26,  1.32s/it]worker  - success: 0.008 - failed to download: 0.992 - failed to resize: 0.000 - images per sec: 39 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1134 - count: 290000worker  - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 39 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1173 - count: 300000worker  - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 39 - count: 10000total   - success: 0.005 - failed to download: 0.995 - failed to resize: 0.000 - images per sec: 1207 - count: 31000032it [04:51,  8.13s/it]worker  - success: 0.037 - failed to download: 0.963 - failed to resize: 0.000 - images per sec: 35 - count: 10000total   - success: 0.006 - failed to download: 0.994 - failed to resize: 0.000 - images per sec: 1133 - count: 320000
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400#issuecomment-1925284544,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437UXT2BFEMJHMEKRWVDYRYMJDAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGI4DINJUGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

I think you probably have more network issues than only the DNS if you have
99% failure
Maybe some misconfiguration of docker or the cloud provider?
On Sat, Feb 3, 2024, 1:10 PM thomas chaton @.***> wrote:

Thanks @rom1504 https://github.com/rom1504 The machine has 32 CPUs, so
I thought it should be fine. I am running inside a docker container, so
having some issues to install knot resolver.
I will keep you updated.
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400#issuecomment-1925304562,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437QSZI5RUQQM3UKCJ4LYRYSKPAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGMYDINJWGI
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hey @rom1504 Any idea what I should be looking for on the docker or cloud provider side?
Also, should I use knot or bind9?

I advise you use knot, it's better for this use case.
For network issues, could be a lot of things but maybe a limit on the
number of open handles/files ?
On Sat, Feb 3, 2024, 1:42 PM thomas chaton @.***> wrote:

Hey @rom1504 https://github.com/rom1504 Any idea what I should be
looking for on the docker or cloud provider side?
Also, should I use knot or bind9?
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400#issuecomment-1925311125,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437RWLMAX4FBNH277RDDYRYWDDAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGMYTCMJSGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks, @rom1504 I will check this out.
I managed to install knot on the host but it isn't visible inside the container and networking seems broken. Have you ever tried?

I am also curious what kind of numbers you get without using knot resolver ?

I didn't try to use docker for img2dataset no. Maybe just use the host?
Usually I get 40-50 image/s/core. With 32 cores that would be 1300-1600
image/s
Knot resolver does not change the speed, it increases the success rate
Should be about 80% for laion400m.
On Sat, Feb 3, 2024, 2:40 PM thomas chaton @.***> wrote:

I am also curious what kind of numbers you get without using knot resolver
?
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400#issuecomment-1925324776,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437RCIZOZDJYI7PHWZDTYRY46DAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGMZDINZXGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hey @rom1504 I am trying to get it working on https://lightning.ai/. Yes, my success rate is far from this. So something is wrong.

Here is the PR I am working on: https://github.com/Lightning-AI/pytorch-lightning/pull/19400

@rom1504 Here is the PR I am working on: https://github.com/Lightning-AI/pytorch-lightning/pull/19400 and the API:
I am trying to make data processing efficient while easy to hack around. Here is the example to download  laion400m. Still need some extra optimizations.
import os
from multiprocessing.pool import ThreadPool
from lightning.data import optimize
from lightning.data.processing.readers import ParquetReader
from lightning.data.processing.image import download_image_with_retry
from lightning.data.processing.utilities import SuppressStdoutStderr
from PIL import Image
from time import sleep

input_dir = "the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta"
parquet_files = [os.path.join(input_dir, f) for f in os.listdir(input_dir) if f.endswith(".parquet")]

def process(row):
    image_id, url, text, height, width, image_license, nsfw, similarity = row
    img, err = download_image_with_retry(0, url, timeout=5)
    if err:
        return None, err

    try:
        return [image_id, img, text, image_license, nsfw, similarity], None
    except Exception:
        return None, err

class Fetcher:

    def __init__(self, max_threads=32):
        self.max_threads = max_threads
        self.stored = 0
        self.skipped = 0

    def __call__(self, df):
        print(self.skipped, self.stored)
        rows = [list(row) for row in df.iter_rows() if row[0] is not None]
        with ThreadPool(self.max_threads) as thread_pool:
            for row, err in thread_pool.imap_unordered(process, rows):
                if err is not None:
                    self.skipped += 1
                    continue

                if row[1] is not None:
                    try:
                        row[1] = Image.open(row[1]).resize((224, 224))
                    except:
                        self.skipped += 1
                        continue
                yield row
                self.stored += 1

optimize(
    fn=Fetcher(max_threads=16),
    inputs=parquet_files,
    output_dir="/teamspace/datasets/laion400m",
    num_workers=os.cpu_count(),
    reader=ParquetReader(num_rows=2048, to_pandas=False),
    chunk_bytes="64MB",
)

Ok. Curious if you reach similar speed as img2dataset and how you'd like to
handle distribution
On Sat, Feb 3, 2024, 7:28 PM thomas chaton @.***> wrote:

@rom1504 https://github.com/rom1504 Here is the PR I am working on:
Lightning-AI/pytorch-lightning#19400
https://github.com/Lightning-AI/pytorch-lightning/pull/19400 and the
API:
I am trying to make data processing efficient while easy to hack around.
Here is the example to download laion400m. Still need some extra
optimizations.
import osfrom multiprocessing.pool import ThreadPoolfrom lightning.data import optimizefrom lightning.data.processing.readers import ParquetReaderfrom lightning.data.processing.image import download_image_with_retryfrom lightning.data.processing.utilities import SuppressStdoutStderrfrom PIL import Imagefrom time import sleep
input_dir = "the-eye.eu/public/AI/cah/laion400m-met-release/laion400m-meta"parquet_files = [os.path.join(input_dir, f) for f in os.listdir(input_dir) if f.endswith(".parquet")]
def process(row):
image_id, url, text, height, width, image_license, nsfw, similarity = row
img, err = download_image_with_retry(0, url, timeout=5)
if err:
return None, err
try:
    return [image_id, img, text, image_license, nsfw, similarity], None
except Exception:
    return None, err

class Fetcher:
def __init__(self, max_threads=32):
    self.max_threads = max_threads
    self.stored = 0
    self.skipped = 0

def __call__(self, df):
    print(self.skipped, self.stored)
    rows = [list(row) for row in df.iter_rows() if row[0] is not None]
    with ThreadPool(self.max_threads) as thread_pool:
        for row, err in thread_pool.imap_unordered(process, rows):
            if err is not None:
                self.skipped += 1
                continue

            if row[1] is not None:
                try:
                    row[1] = Image.open(row[1]).resize((224, 224))
                except:
                    self.skipped += 1
                    continue
            yield row
            self.stored += 1

optimize(
fn=Fetcher(max_threads=16),
inputs=parquet_files,
output_dir="/teamspace/datasets/laion400m",
num_workers=os.cpu_count(),
reader=ParquetReader(num_rows=2048, to_pandas=False),
chunk_bytes="64MB",
)
—
Reply to this email directly, view it on GitHub
https://github.com/rom1504/img2dataset/issues/400#issuecomment-1925422636,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR437ST3YPBBFTDOR4FJKTYRZ6TXAVCNFSM6AAAAABCX6KGQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGQZDENRTGY
.
You are receiving this because you were mentioned.Message ID:
@.***>

It seemed Image downloading speed were quite similar. But I need to be more principled and collect the same metrics to have a more educated comparaison.
But first, I need to resolve the downloading speed and success rate behind so low.
But the StreamingDataset is faster than Webdataset though.  It contains everything, python deps, code, data, etc...

The distribution is already fully handled by the optimize and map operators. Check this example: https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset?view=public&section=data+processing
Example to tokenized SlimPajama.
import json
from pathlib import Path
import zstandard as zstd
from lightning.data import optimize
from tokenizer import Tokenizer
from functools import partial
from lightning_sdk import Machine

# 1. Function to tokenize the text contained within the Slimpajama files
def tokenize_fn(filepath, tokenizer=None):
    with zstd.open(open(filepath, "rb"), "rt", encoding="utf-8") as f:
        for row in f:
            text = json.loads(row)["text"]
            if json.loads(row)["meta"]["redpajama_set_name"] == "RedPajamaGithub":
                continue  # exclude the GitHub data since it overlaps with starcoder
            text_ids = tokenizer.encode(text, bos=False, eos=True)
            yield text_ids

# 2. Generate the inputs (we are going to optimize all the compressed json files from SlimPajama dataset)
input_dir = "/teamspace/studios/SlimPajama_Dataset/data/train"
inputs = [str(file) for file in Path(input_dir).rglob("*.jsonl.zst")]

# 3. Store the optimized data wherever you want under "/teamspace/datasets" or "/teamspace/s3_connections"
outputs = optimize(
    fn=partial(tokenize_fn, tokenizer=Tokenizer("./checkpoints/Llama-2-7b-hf")), # Note: You can use HF tokenizer or any others
    inputs=inputs,
    output_dir="/teamspace/datasets/slimpajama/train/",
    chunk_size=(2049 * 8012),
    num_nodes=16,
    machine=Machine.DATA_PREP, # use 32 CPU machine
)

This remotely process the full dataset over 16 nodes.

Hey @rom1504 I am able to get 1.1k images/sec.
I think I have a version of knot resolver that works. I am also using http2 from httpx client and I sorted to parquet files by URL to hopefully help slightly the DNS resolving.
But the ratio of success is around 60%, so quite far from yours though. I will try again img2dataset. There is possibly something with docker not well configured.
Best,
T.C

Be careful with sorting the urls as you risk to dos the hosts. I had randomly shuffled them in laion datasets to mitigate this.
Some people recently have had some success by calling knot with all unique domains to get its cache ready.
Usually I didn't hit issues with DNS when using knot though. Issues only happens in some environments with restricted DNS setup

But the ratio of success is around 60%, so quite far from yours though.

You can log the errors to try and understand what the cause is. In img2dataset there is a wandb table for it.

Hey @rom1504 I am able to get 1.1k images/sec.

Nice! How many cores are you using?

Be careful with sorting the urls as you risk to dos the hosts. I had randomly shuffled them in laion datasets to mitigate this.

Interesting. Yes, I didn't think of that. Good call !

Some people recently have had some success by calling knot with all unique domains to get its cache ready.

This is a good idea. I will see if there is a simple way for to add support for this.

Nice! How many cores are you using?

I am using a 32 CPU machine, so slightly lower than what you told me to expect.

# main ones
- [Errno 101] Network is unreachable,
- [Errno 99] Cannot assign requested address
- [Errno -2] Name or service not known

# the rest
- [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'site.aimbulance.com'. (_ssl.c:997)

Interestingly, the ratio with img2dataset is quite lower:
worker  - success: 0.105 - failed to download: 0.894 - failed to resize: 0.001 - images per sec: 308 - count: 10000
total   - success: 0.082 - failed to download: 0.918 - failed to resize: 0.000 - images per sec: 2979 - count: 96608
worker  - success: 0.146 - failed to download: 0.854 - failed to resize: 0.000 - images per sec: 313 - count: 10000
total   - success: 0.088 - failed to download: 0.912 - failed to resize: 0.000 - images per sec: 3288 - count: 106608
worker  - success: 0.128 - failed to download: 0.872 - failed to resize: 0.000 - images per sec: 300 - count: 10000
total   - success: 0.091 - failed to download: 0.909 - failed to resize: 0.000 - images per sec: 3497 - count: 116608
worker  - success: 0.124 - failed to download: 0.876 - failed to resize: 0.000 - images per sec: 311 - count: 10000
total   - success: 0.094 - failed to download: 0.906 - failed to resize: 0.000 - images per sec: 3797 - count: 126608
worker  - success: 0.174 - failed to download: 0.825 - failed to resize: 0.001 - images per sec: 343 - count: 10000
total   - success: 0.100 - failed to download: 0.900 - failed to resize: 0.000 - images per sec: 4097 - count: 136608
worker  - success: 0.159 - failed to download: 0.840 - failed to resize: 0.001 - images per sec: 178 - count: 5536
total   - success: 0.102 - failed to download: 0.898 - failed to resize: 0.000 - images per sec: 4263 - count: 142144
worker  - success: 0.090 - failed to download: 0.909 - failed to resize: 0.001 - images per sec: 317 - count: 10000
total   - success: 0.101 - failed to download: 0.898 - failed to resize: 0.000 - images per sec: 4563 - count: 152144
worker  - success: 0.149 - failed to download: 0.851 - failed to resize: 0.000 - images per sec: 313 - count: 10000
total   - success: 0.104 - failed to download: 0.896 - failed to resize: 0.000 - images per sec: 4863 - count: 162144
worker  - success: 0.082 - failed to download: 0.918 - failed to resize: 0.000 - images per sec: 305 - count: 10000
total   - success: 0.103 - failed to download: 0.897 - failed to resize: 0.000 - images per sec: 5163 - count: 172144
worker  - success: 0.120 - failed to download: 0.880 - failed to resize: 0.000 - images per sec: 304 - count: 10000
total   - success: 0.104 - failed to download: 0.896 - failed to resize: 0.000 - images per sec: 5463 - count: 182144
worker  - success: 0.102 - failed to download: 0.897 - failed to resize: 0.001 - images per sec: 316 - count: 10000
total   - success: 0.104 - failed to download: 0.896 - failed to resize: 0.000 - images per sec: 5763 - count: 192144
worker  - success: 0.099 - failed to download: 0.901 - failed to resize: 0.000 - images per sec: 305 - count: 10000
total   - success: 0.104 - failed to download: 0.896 - failed to resize: 0.000 - images per sec: 6063 - count: 202144
worker  - success: 0.194 - failed to download: 0.806 - failed to resize: 0.000 - images per sec: 318 - count: 10000
total   - success: 0.108 - failed to download: 0.892 - failed to resize: 0.000 - images per sec: 6363 - count: 212144
worker  - success: 0.152 - failed to download: 0.848 - failed to resize: 0.000 - images per sec: 308 - count: 10000
total   - success: 0.110 - failed to download: 0.890 - failed to resize: 0.000 - images per sec: 6644 - count: 222144

{
    "count": 10000,
    "successes": 900,
    "failed_to_download": 9093,
    "failed_to_resize": 7,
    "duration": 31.51988196372986,
    "start_time":<PHONE_NUMBER>.7824914,
    "end_time":<PHONE_NUMBER>.3023734,
    "status_dict": {
        "<urlopen error [Errno -2] Name or service not known>": 996,
        "<urlopen error [Errno -3] Temporary failure in name resolution>": 31,
        "success": 900,
        "Image decoding error": 7,
        "HTTP Error 404: Not Found": 105,
        "timed out": 1,
        "HTTP Error 403: Forbidden": 23,
        "<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for '0.realtorpage.io'. (_ssl.c:997)>": 1,
        "<urlopen error [Errno 99] Cannot assign requested address>": 7936
    }
}```

Hey @rom1504 I found this interesting issue: https://github.com/pola-rs/polars/issues/14358. I definitely need to add profiling.

Hey @rom1504, I have made some progress today. Still some work left.
I am getting roughly a 68% success ratio with 60 images/seconds/core.
 This is my first test run. I will keep you updated.

Sorry to bother you. Could you tell me how to download laion400M dataset? I use this code try to download:img2dataset --url_list laion400m-meta --input_format "parquet" --url_col "URL" --caption_col "TEXT" --output_format webdataset --output_folder laion400m-data --processes_count 16 --thread_count 128 --image_size 256 --save_additional_columns '["NSFW","similarity","LICENSE"]' --enable_wandb True, but sth wrong happened.

Hey @SomnusQue Here is the full blogpost explaining how to download the dataset: lightning.ai/lightning-ai/studios/download-stream-400m-images-text~01hg0zg8fyybp7p1sma6g9dkzm.
@rom1504 I would appreciate if you could have a read and give me your thoughts.


Wizard in Metro 4

What do you think about the new version of Wizard in Metro 4?
https://metroui.org.ua/v4/wizard.html

Neat, but i think there's one feature missing, the possibility to go back on a certain step by clicking on its number/vertical bar


Underscore issue on column names

Hi,
There is an issue with column mapping with underscore character.
If column name contains underscore, Query function does not fill that column.
Example:
var rows = connection.Query<SomeRow>("SELECT x1, some, other, columns, but_i_am_not_cool FROM hede")
after this:
rows[0].x1 //OK
rows[0].some //OK
rows[0].other //OK
rows[0].columns//OK
rows[0].but_i_am_not_cool // NOT OK, always null

Isn't it related to Dapper ?
Dapper.DefaultTypeMap.MatchNamesWithUnderscores = true;


add channels for Roku TVs

I wasn't able to get this working in the tests (I'm pretty new to python 🙁), but it seems like it should work.

Relevant docs: https://sdkdocs.roku.com/display/sdkdoc/External+Control+API#ExternalControlAPI-query/tv-channels

@bah2830 Is there anything I can do to help this get merged?

@bah2830 would you have some time to review this? it's been about 15 months now.


Cur VSTS header

https://docs.microsoft.com/en-us/azure/devops/artifacts/_img/delete/delete-maven-package.png?view=azure-devops
please, cut header from VSTS age.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 31a8d31b-229b-7383-1cc0-f3d7c65eca84
Version Independent ID: 70ba7acf-adae-e1b7-a42d-3f1344c5293c
Content: Delete and recover packages
Content Source: docs/artifacts/how-to/delete-and-recover-packages.md
Product: devops
GitHub Login: @elbatk
Microsoft Alias: elbatk

I have updated the screen shot and you should be able to see the update within 24 hours. If you have anymore feedback, please feel free to reach out or create a new issue


Protect by id and type of entity

This PR introduces an implementation of the
public boolean hasPermission(Authentication authentication, Serializable targetId, String targetType, Object permission)

which is based on the ID and the type of a target entity.
The implementation provided here searches for a matching DAO, retrieves the entity from the database and simply calls the (existing) implementation of hasPermission with the object.
Two methods in the MapService and the LayerGroupService now make use of the new implementation.
Please review

Although the title seems to be missing something, this PR adds a nice new flexibility!
I like it, :+1:

Pleas give this PR a better title and also please take the time to think about the longer remark I added to the diff.
Regardless of whether you take the suggestion, this is good to go IMO. Thanks @buehner. Please merge at will (taking or ignoring my suggestion).

Yes. You're right @marcjansen
But this will probably never be a "real" problem as we only need a DAO to call findById (which will most likely always be the same implementation).
So i will merge this PR now and open a new issue to adress your proposal.

I just opened #187


Getter is required for static data, but could be omited

class Find
  include Supa::Representable

  define do
    namespace :jsonapi do
      attribute :version, getter: -> { '1.0' } # now
      attribute :version, '1.0'  # preferred
    end
  end
end

We take care of this feature in https://github.com/distribusion/supa


feat(forms): Introduce useFormikEnhancedContext.

@doniyor2109 why don't you use useFormikContext<TValues>() as FormikContextTypeEnhanced? we will not need that after https://github.com/jaredpalmer/formik/pull/2323 anyways


key :key not found in [store: PhoenixLiveSession, pub_sub, Sage.PubSub, signing_salt.....

Hello.
Just following the instructions to use PhoenixLiveSession and I'm getting the error in the title key :key not found. It is unclear to me whether I should simply add the :key or if it is not required and I've done something wrong :-).
I'm using
  {:phoenix, "~> 1.5.7"},
  {:phoenix_live_view, "~> 0.15.0"},
  {:phoenix_live_session, "~> 0.1"}

cheers

Could you share the code you’ve used for configuring Live Sessions and the exact error message?

Hi my endpoint.ex is:
@session_options [
   store: PhoenixLiveSession,
   pub_sub: Sage.PubSub,
   signing_salt: "2Ytbl+mr"
 ]

and the error I get is....

== Compilation error in file lib/sage_web/endpoint.ex ==
** (KeyError) key :key not found in: [store: PhoenixLiveSession, pub_sub: Sage.PubSub, signing_salt: "2Ytbl+mr"]
    (elixir 1.11.2) lib/keyword.ex:420: Keyword.fetch!/2
    (phoenix 1.5.7) lib/phoenix/socket/transport.ex:270: Phoenix.Socket.Transport.init_session/1
    (phoenix 1.5.7) lib/phoenix/socket/transport.ex:255: anonymous fn/1 in Phoenix.Socket.Transport.load_config/1
    (elixir 1.11.2) lib/enum.ex:1399: Enum."-map/2-lists^map/1-0-"/2
    (phoenix 1.5.7) lib/phoenix/socket/transport.ex:250: Phoenix.Socket.Transport.load_config/1
    (phoenix 1.5.7) lib/phoenix/endpoint.ex:638: Phoenix.Endpoint.socket_paths/4
    (phoenix 1.5.7) lib/phoenix/endpoint.ex:591: anonymous fn/3 in Phoenix.Endpoint."MACRO-__before_compile__"/2
    (elixir 1.11.2) lib/enum.ex:2181: Enum."-reduce/3-lists^foldl/2-0-"/3
    (phoenix 1.5.7) expanding macro: Phoenix.Endpoint.__before_compile__/1
    lib/sage_web/endpoint.ex:1: SageWeb.Endpoint (module)
    (elixir 1.11.2) lib/kernel/parallel_compiler.ex:314: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7

Adding :key appears to get it to work but I'm worried this is masking an issue. cheers

Yes, adding :key is necessary when defining a session, this should be clearer in the README. I’ll fix it there, thanks for pointing it out!

No problem at all - thanks for the awesome library.


First example ?

Hi, sorry for the inconvenience, but I tried to test this lib and the Mastodon API but I didn't succeed and don't know why.
Is there any simple runnable example to start with ?
For information, I've got this error while trying to do the app.connect(server_url, user_email) :
Failed to find token in storage & no code found in URL parameters.
  get_token @ libodon.js:144
  get_registration.then.reg @ libodon.js:16
libodon.js:24 Uncaught (in promise) Object {error: "failed connection"}
  get_registration.then.then.error @ libodon.js:24
libodon.js:25 Uncaught (in promise) ReferenceError: reg is not defined
  get_registration.then.then.error @ libodon.js:25

Thanks

I found a bug and fixed it. The code hasn't been tested that much and is still only half-finished, but feel free to use and/or improve it :)


Don't fail when a tasklog has no tasks

Technically, there's no guarantee that a task log is associated with at
least one task.  When jobs get deleted, tasklogs with no tasks sometimes
do happen.

Coverage decreased (-0.64%) to 64.99% when pulling db84953924f171ca7e4392d8e35733bb2d8da8f7 on guidow:logs_with_no_tasks into b186a96ecc8da74e872a7a02667f82cbc12530a9 on pyfarm:master.

Looks good, merging since this would not impact the tests.


fix(CI): add git username

Description
Adds git username to avoid error when back merging during release job.

Sanity Tests Execution Failed.

I'm curious, what's the error we're resolving here? the specific git username shouldn't really matter since permissions are based on the PAT. is it mad about the spaces?

@anthony9187 I found that when testing the merge fails if the username does not match the username that the PAT was created with. We may be able to remove the config email however @ImCoolNowRight.

I'm curious, what's the error we're resolving here? the specific git username shouldn't really matter since permissions are based on the PAT. is it mad about the spaces?

@anthony9187 This is the error we were encountering during the build: https://github.com/rdkcentral/Lightning-UI-Components/actions/runs/5349179450/jobs/9700088743

Sanity Tests Execution Passed.


Opening minimified JS files makes VSCode unusable for minutes

VSCode Version: 1.49.0
OS Version: Windows 10

I have a project that mixes PowerShell/JS/CSS/HTML - https://github.com/EvotecIT/PSWriteHTML. In this project I mainly work with PowerShell but often have to switch to JS code to check something or commit minified JS to GitHub. This is where the problems starts. Once I switch to JS everything starts crawling, PowerShell extension "hides" and recovers after minutes.
Originally reported in PS Extension: https://github.com/PowerShell/vscode-powershell/issues/2920
This is behaving similarly bad when I go to Source Control tab and it tries to do comparison, making it very hard to commit 450KB file to github. My biggest file is 948KB

Steps to Reproduce:

Open PowerShell script, make sure PS extension does the highlighting and is visible in right bottom corner
Switch to JS file of 450KB size
Notice how VSCode becomes unresponsive
Switch back to PowerShell file, notice how in the right bottom corner the 5.1 version is gone, and colors of PowerShell changed meaning whole extension is not responding

Same thing happens if you try to commit min.js to git. Once it opens up .js you're not able to commit anything until it's done doing something in the background.

Does this issue occur when all extensions are disabled?: Yes/No

It seems it's extension fault Bracket Pair Colorizer 2.


Add JVM metrics for total heap memory usage from MemoryMXBean

I opened https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/9859 and was told to open an issue here in the specification in order to propose this change.
Is your feature request related to a problem? Please describe.
The heap memory JVM metrics are currently being obtained from MemoryPoolMXBean.
I'm having difficulty calculating the correct maximum total values from the current jvm.memory.* metrics because they're divided by the memory pool attribute. For example, if I want to calculate the maximum total heap memory usage, it's possible to sum up the values from:
max:process.runtime.jvm.memory.usage{type:heap, pool:eden_space} + max:process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} + max:process.runtime.jvm.memory.usage{type:heap, pool:survivor_space}

But if a service has multiple JVM instances, then it becomes difficult to distinguish between the JVM instances and this query will sum across all JVM instances in this service.
If there was another metric for process.runtime.jvm.total.memory.usage (from MemoryMXBean), then we could query max:process.runtime.jvm.total.memory.usage{type:heap} for the correct total maximum heap memory across all JVM instances in the same service.
Describe the solution you'd like
I'd like to add the following heap memory JVM metrics from MemoryMXBean:

process.runtime.jvm.total.memory.usage
process.runtime.jvm.total.memory.committed
process.runtime.jvm.total.memory.init
process.runtime.jvm.total.memory.limit

I can help open a PR for this if the feature request is approved.
Describe alternatives you've considered
No response
Additional context
No response

@jack-berg The query you wrote is possible in the query language I'm using, but isolating the instance isn't possible in all of our use cases. For more context, we're making a universal dashboard for customers that should ideally work for any deployment out of the box, so we don't know what unique attributes can be used as a grouping key. We can expect customers to set some attributes set on the metrics to make sure they are distinct, but we don't know in advance what they would be.
If we had the total metrics from MemoryMXBean, then we could display accurate values for averages, maxes, mins, and sums across JVM instances/services even when the instance can't be isolated.
For example, we want to calculate the maximum total heap memory usage using the formula from above:
max:process.runtime.jvm.memory.usage{type:heap, pool:eden_space} + max:process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} + max:process.runtime.jvm.memory.usage{type:heap, pool:survivor_space}

But we can't isolate the instance in advance, so this formula could end up adding values from different JVM instances, which wouldn't represent the correct total maximum heap memory usage.
If we could query max:process.runtime.jvm.total.memory.usage{type:heap} instead, then this would display the correct value across JVM instances/services without requiring the customer to manually filter on their distinct attributes in order to get the correct value.

I don't understand why the tooling doesn't allow you to query for type:heap (i.e. ignore the pool attribute) on the existing process.runtime.jvm.memory.usage metric. Why does the presence of the additional pool attribute cause issues?
Spatial reaggregation is a fundamental part of the metrics data model. Both SDKs and the collector have tools to drop unwanted attributes (i.e. views for the SDK and the metrictransformprocessor for the collector). Even the versioning and stability requirements, which are particularly strict around metric attributes, allow adding new attributes to existing metrics that do not "break apart".
What you're asking for is essentially a special metric view with a reduced set of attributes, and would result in the same information being sent up in two different ways. Semantic conventions should not do this type of thing.

even when the instance can't be isolated.
But we can't isolate the instance in advance, so this formula could end up adding values from different JVM instances,

Maybe this is the crux of your issue. Why can't the instance be isolated? Are you trying to provide a dashboard which works when regardless of service.instance.id is present or not? If so, consider making service.instance.id a requirement for this feature. Many aspects of JVM runtime conventions break down if there isn't an attribute like service.instance.id which uniquely identifies the instance within a particular service. When service.instance.id is not present, the single write principle is being violated, since two unique JVMs can be emitting time series with the same identity.

@jack-berg The presence of the additional pool attribute doesn't cause issues, but the metric values are divided into each memory pool attribute which results in a different total metric value. For example:
From MemoryMXBean, process.runtime.jvm.memory.usage{type:heap} has a data point with value 100.
From MemoryPoolMXBean, This same metric would be reported as:

process.runtime.jvm.memory.usage{type:heap, pool:eden_space} data point value = 50
process.runtime.jvm.memory.usage{type:heap, pool:tenured_gen} data point value = 25
process.runtime.jvm.memory.usage{type:heap, pool:survivor_space} data point value = 25

This causes issues because we then need to aggregate the pool values somewhere in our pipeline. I agree that it would be ideal if the instance id could be guaranteed, but we can't guarantee this since it's intended for a universal dashboard that should work with many different kinds of deployments.
The metricstransformprocessor seemed like it would work for aggregating the pool values, but then we realized that it wouldn't work for Gateway deployments because it would require all of the pool values to be sent to the same Collector instance. This could be done with the loadbalancingexporter, but it would add a lot of complexity and metrics are still in development.
Would it be possible to add the total value from MemoryMXBean as an attribute on the existing metrics for usage, committed, init, and limit if adding new metrics isn't possible?

This causes issues because we then need to aggregate the pool values somewhere in our pipeline. I agree that it would be ideal if the instance id could be guaranteed, but we can't guarantee this since it's intended for a universal dashboard that should work with many different kinds of deployments.

You can reject the data or disable the dashboard if service.instance.id isn't included. I think doing so is reasonable because without it, there's a high probability the single writer principle is being violated. See #311 for an issue attempting to make service.instance.id always present.

Would it be possible to add the total value from MemoryMXBean as an attribute on the existing metrics for usage, committed, init, and limit if adding new metrics isn't possible?

No, the memory usage is a high cardinality value and metric attributes need to be limited to low cardinality values. Also, attributes are for describing the data, not for encoding values.
I don't think this is an issue with JVM memory metrics or JVM metrics in particular, but actually a broader question of ensuring there is always way to do spatial reaggregation in a backend, which requires service.instance.id. I would recommend @open-telemetry/specs-semconv-maintainers close this particular issue but reference the use case on #311.

Closing this as no activity and I agree with the comments above from @jack-berg. @liustanley feel free to follow #311 or post your concerns there as well.


improve scripting/config for postgres

This looked good when I read through the code.  Doing some post-merge testing on it.

So...I've run though this a couple of times.  It seems to do the right thing.
There is an unwritten requirement (I think) that it's being run from within a checked-out source tree, and OSCARS has been compiled and installed in that tree.  I'm not sure if those conditions would be met in a real production deployment, as opposed to having the JAR files, etc. being packaged up and distributed, as is done for example with many of the NEG tools.  The trend with the automation group is towards packaging, which enables INF (or whomever) to push out new versions of packages using existing OS mechanisms, rather than requiring a developer to do git pull, etc. on a production machine.


Config Server Client Security

I have briefly discussed the problem as I was running into on https://github.com/spring-cloud/spring-cloud-config/issues/84.  During that time, I was missing the keystore in the client side but it does not solve my problem when I try to deploy Config Server into production.
Background
I am deploying the Config Server as a company wide shared configuration service (for production and non-production environment by using the profile annotation).  Since all the production passwords are encrypted using the Config Server /encrypt endpoint, it is not OK to allow developers have direct access to the Config Server without authentication.
I was planning to use the basic HTTP authentication with the Config Server and its bootstrap.yml looks like the following:

encrypt:
  failOnError: false
  keyStore:
    location: classpath:config-server.jks
    alias: config-server
    password: f7hkjd764jAl

and the configserver.yml looks like the following, where the password was encrypted using it's own /encrypt endpoint with the config-server.jks keystore:

security:
  user:
    name: foobar
    password: "{cipher}AQAUtSkpM24ClZ81a/696J8Fdr2gkZUSsT2A4Zr4RBsRx4CNpWO+/f0Xs2Cazusjm3sRRABCgdkU6EZWWnrul/Up6DsgW5F3tzEgFMv51hZ4kw8hdkOXkZhohuOAIvMz6ZK4/d4kiDzRPxxWextAmvc+umCOfgRoFxFzi86wfvqO+WpAo41aOMRBC16kdycjs2zq8PG1FiZ631a11hNd0VB8N55kshXq0pTSoXLu3d7r2BwEogdhdrbeeuZ9Vfr1419aVJAxFmEIwni3pdn3Yj8bk/NbL0VQAXJPSlrbocNh1bWkSguX3g0pA44YqEIhwjQwsX3D10q1LIJrkcjEmKosydyuU3od46GDMMQDBaOETA7nKorCjFDR/ppi02TS59s="

On the client side, the bootstrap.yml looks like the following:

spring:
  application:
    itemName: my-service
  cloud:
    config:
      uri: http://config.server.internal:8080
      failFast: true
      username: foobar
      password: "{cipher}AQAUtSkpM24ClZ81a/696J8Fdr2gkZUSsT2A4Zr4RBsRx4CNpWO+/f0Xs2Cazusjm3sRRABCgdkU6EZWWnrul/Up6DsgW5F3tzEgFMv51hZ4kw8hdkOXkZhohuOAIvMz6ZK4/d4kiDzRPxxWextAmvc+umCOfgRoFxFzi86wfvqO+WpAo41aOMRBC16kdycjs2zq8PG1FiZ631a11hNd0VB8N55kshXq0pTSoXLu3d7r2BwEogdhdrbeeuZ9Vfr1419aVJAxFmEIwni3pdn3Yj8bk/NbL0VQAXJPSlrbocNh1bWkSguX3g0pA44YqEIhwjQwsX3D10q1LIJrkcjEmKosydyuU3od46GDMMQDBaOETA7nKorCjFDR/ppi02TS59s="
encrypt:
  failOnError: false
  keyStore:
    location: classpath:config-server.jks
    alias: config-server
    password: f7hkjd764jAl

Now, my problem is:  The developers have full access to the server side key-pair.  In other words, the developers can decrypt the {cipher} password from the properties files by using the keystore.  It is completely defeated the purpose of server side encryption.
Questions

Is it possible using a different keypair for config server login credential?
I may be still do it completely wrong up there.  Any correction is helpful since the current document is very unclear on how to use encrypted password for http authentication.

You can certainly use a different keypair for the config server login credential on the client side (not the server side).  Putting the production config server password in bootstrap.yml checked into source control, is probably not the way to go about securing the config server.  On the production server you could use an environment variable (SPRING_CLOUD_CONFIG_PASSWORD) or use a separate keypair (or encrypt.key) on the client side to encrypt/decrypt the password for config server clients.

I think I got it.
The client side needs to do the decryption on the password before sending over to the Config Server side.  So I should be using a different set of keypair for the client side.
And yes, the client side password will be assigned onto the production server as env variable so nothing will check into the source control.